Reproducibility of AOD Algorithm: An Experimental evaluation for Key-Predictors Identification

INTRODUCTION: Today surveillance systems are widespread across the globe for monitoring of various activities. Abandoned Object Detection (AOD) and identifying its location is one of them. In this paper, we evaluated the reproducibility of an existing AOD algorithm on benchmark video datasets. OBJECTIVES: The purpose of the study is to identify the key predictors for developing a generalized AOD algorithm. METHODS: The algorithm selection is performed by a detailed exploration of repositories through various research questions (RQs). RESULTS: After the study video summarization, Correct Detection Rate (CDR), generalized Region of Interest (ROI), background learning, and interaction factor considered for enhancing the AOD algorithm. CONCLUSION: Identification of suspiciousness has various measures depending upon perception, on the basis of results explored the existing algorithm can be improved using key-predictors with observational parameters.


Introduction
As per the literature in past decades video surveillance has emerged as a globally acceptable high-end security device for suspicious activity identification. There are various categories of suspicious activities among them the authors focus on Abandoned Object Detection (AOD). An abandoned object is an object left by a person in public premises. Detection of abandoned object is important because as per the reports public places are targeted for attacks through bomb blasts. Also, it was observed that sometimes a person left object by mistake at public premises. Therefore, identifying a person motive behind the object to be abandoned is also important to enhance the surveillance security. In literature, many algorithms exist for AOD, however very few for the motive identification * Corresponding author. Email: monika.usict.099164@ipu.ac.in of a person. Therefore, in this paper, we focus on improving existing AOD methods [1][2][3][4][5] by analysing them through in-depth filtration using RQs. These RQs helps in the selection of existing AOD method using which the detail study to improve an algorithm using key predictors has been identified. To accomplish it the existing methods are tested on benchmark AOD datasets stated in section 3. These datasets are created by considering different aspects of security through surveillance. This process will help us to improve the existing AOD method and give us a direction towards developing a generalized AOD algorithm. In this paper, the prime focus is on the identification of key predictors to improve the existing AOD algorithm [6,7]. To achieve this objective, we explored so many repositories and research sources. It is considered to test the reproducibility of existing methods. The flow of the paper and the important phases done to accomplish the work has been shown in figure 1. The process is depicted through an incremental arrow as backtracking is refuted. The process starts with putting the challenges found which has been listed below as major observation. On these challenges, RQs have been posed which has been discussed in detail in Section 2. Further, the method selection is done on various filters which have been depicted through Table 2. The selected method has been tested in Section 4. The key-predictors derived from the experiments have been discussed in detail in Section 5.

EAI Endorsed Transactions
From this exploration, we discovered that a wide range of the existent methods for AOD is not available for use. The discovered method found for testing the reproducibility is proposed by MATLAB corporation for testing. The major observations found before testing the reproducibility have been listed as follows: -• The most prominent frame extraction method observed is insignificantly lengthy in processing. • The MATLAB AOD is efficiently reproducible only on its own example video dataset, however, it generated results on a few samples of benchmark AOD datasets. • Uses a fixed ROI.
• No parameter to identify accuracy and efficiency.
• Other aspects like the group behavior analysis has not been considered.
Therefore, in this paper, various types of benchmark AOD datasets have been tested on the existing AOD method to identify the key predictors for the proposed AOD method. Further, the paper is composed of four more sections to explain the work in detail. The next section gives a glimpse of method selection for testing the benchmark datasets. In the third section, the AOD method explained in detail. In the fourth section, the experimental analysis performed to evaluate the method's performance. In the next section, identified key predictors has been discussed. Lastly, the work has been concluded with concluding remarks and references.

AOD Method Selection
Before initiating the process of identification of key predictors. An in-depth method for the exploration of research sources and repositories has been performed through various Research Questions (RQs). The RQs are posed while keeping in mind that the selected existing AOD method is viable for use. Therefore, the RQs are as follows: -RQ1. Has the chosen repository is reputed? RQ2. Is it a copyrighted method/algorithm? RQ3. Has the stated method exists to use? RQ4. Is the method being reproducible on another dataset? Repositories/ research sources explored for this purpose are shown below in Table 1 with the weblinks. The first two sources have been explored for the programming method and code availability. The next three sources have been used to identify respective research papers for algorithms and their respective codes. The last source has been used for exploring AOD methods in non-reputed sources. Further through Table 2, the repositories are filtered using the RQs. Reproducibility of AOD Algorithm: An Experimental evaluation for Key-Predictors Identification 3 made available by MATLAB corporation. The reason for the selection of the method is that it is part of a reputed organization and copyrighted as well. On the basis of RQs, the experimental analysis of the work to test its reproducibility has been performed in section 4.

AOD Method
In literature various algorithms/methods have been seen regarding identification/detection of abandoned objects, through the RQs existing AOD algorithm has been selected. In this section in order to ensure the reproducibility of the method, we explored various sources to test the same code on benchmark datasets. This section is further divided into two categories as the AOD method and benchmark dataset i.e. section 3.1 and section 3.2 respectively.

AOD Method
As per the RQs, the chosen existing AOD algorithm has been performed on all phases of AOD. The block definition of the AOD algorithm has been shown in figure 2 [19]. The block diagram shows various phases of the framework as components of the AOD. The first block shows the input sample used to perform AOD i.e. video sample of AOD which has a frame rate of 240X360 at 30 frames per second. The next block shows the Region of Interest (ROI) extraction unit where two different ROIs for X and Y coordinates are used as input to the block. The parallel process to it is the construction of ROI block which continuously maps the two ROI on the video during processing. In the next step of AOD, the conversion of input frames from RGB format to Y'CbCr is performed; and simultaneously the step function is executed to trigger the stored background block. In this phase, the converted image and step function together executed to generate the detection results for the next phase. The next block i.e. 'Detection' shows the vision-based methods [10][11][12][13][14] used for the detection of AOD. To yield more accuracy in AOD, various predefined MATLAB library functions of the 'Computer Vision' package has been used. The parameters considered in the blocks are chosen on various criteria's of AOD such that most of the cases can be covered. The parameters are chosen to give on various ports of the block to perform the desired tasks. By considering various factors, we implemented this algorithm on various benchmark datasets and the corresponding experimental results and analysis of the same is performed in the next section.

Benchmark Dataset
It is also called as a public dataset as it is freely available to use by the researchers. These are said as 'Benchmark Dataset' because it is verified by the research groups of long-standing repute.

Experimental Analysis
In this section MATLAB, the AOD method is implemented on various benchmark AOD datasets stated in section 3.2. The experimental work performed in MATLAB 2013a with 8 GB of RAM on Intel 7 processor of 8 th generation. The execution results of the AOD algorithm that has been performed on the AOD benchmark datasets are shown in Table 3. Table 3 shows the execution results generated through the 'Display Results' block. As the AOD process runs until the whole video file is not executed. Therefore, we considered displaying an initial frame and the last frame snippet of the video during algorithm processing. The snippets of AVSS2007 dataset are not included because the algorithm uses initial frames for background learning and the initial frame of AVSS2007 dataset includes a description of it. So, the background learning process in it not initiated successfully. Due to this algorithm failed on AVSS2007 dataset and the processing results of the AOD method on AVSS2007 is not listed in table 3. The snippets display a count of AOD objects detected and the red coloured small boxes as detected abandoned objects.

Observational parameters
Observational parameters help to identify the issues and their severity. Therefore, Table 3  The parameters are selected to satisfy the minimum criteria for testing the reproducibility of the existing algorithm. The parameters and evaluation criteria have been given below.

Algorithm Reproducibility
This parameter will define whether the algorithm is successfully implemented on the benchmark dataset or not. It is evaluated through the equation 1 given below.

Background Learning
This parameter defines the background learning capability of an algorithm. As you can see that in the case of AVSS it is 0 because it fails to learn the BGL. This parameter is defined in two categories that whether BGL is learned by the algorithm or not. The parameter criteria are shown below.
Learning Learned Not Learned BGL 1 0 As you can observe from the table that in case of AVSS its 0 because it fails to learn the BGL and hence the whole algorithm fails to work on it.

Interaction Factor
An object in a surrounding has some level of interaction with others. For example, a person interacted with a person and drop the belonging next to that person and move towards the ticket counter. So, in this case, the interaction and relativity of a person towards the object is positive. However, if a person left an object alone and move towards "Exit" then the probability is high that interaction and relativity of that object is negative. The criteria for IF are given below.

Interaction
Positive Negative IF 0 1

Area Coverage
The last parameter considered to be observed during algorithm processing is AR. The reason being in case of video processing ROIs are used highly to avoid unnecessary computation. However, in the case of AOD while testing the reproducibility it has been observed that the ROI regions left some area and due to it the AODs are not completely detected. The criteria to define the parameter at different levels is given below.

Coverage Null Less More
The criteria are null, less, more and all having values 0,1,2,3 respectively. A value is Null if no AOD found in the complete scene, less if AOD is outside the ROI. A value for AC is More if AOD is within and outside the ROI. The last criteria are observed if all the AOD is in ROI. In criteria, Less and More of IF the AOD detected may be correct/incorrect. Using all the criterion the Table 3 Observations Column has been obtained.

Result Analysis
Further, for more clarity and ease in the analysis of key predictors, we computed various factors that have been shown in Table 4. In Table 4, the initial columns give details of the dataset and its sample type. Further predefined elements identified which are resolution, the total number of frames, frame rate and actual abandoned objects in the sample of the dataset. Whereas, the processed elements are those generated after applying the algorithm. So, the processed elements are AODs, Actual AODs, and CDR which are abandoned objects detected by the algorithm, actual abandoned object detected in the sample and correct detection rate respectively. The formula used to compute CDR is given through equation 2 below. (2) Using this equation, we tried to calculate the correct detection rate of algorithm overall detected objects in the AOD category. And to accomplish it the overall division of AODs is performed on actual AODs detected over actual AOs predefined. It is multiplied by 100 to get the CDR in percentage form. This formula is derived because it gives detection rate more accurately as compared to existing methods in the literature. Table 4. CDR computation on the AOD dataset.

Key-Predictors Identification
Key-predictors are significant elements which helps to improve the algorithm efficiency. Key-predictors are identified by the issues raised by experimental analysis. In order to identify the issues and its related key-predictors, experimental analysis has been analyzed with the observational parameters. The issues identified with the key-predictors is depicted through figure 3.  The figure shows that for automated abandoned object detection, the video feed has issues such as different resolution and frame rate, similarly ROI can be different, and a person in the environment has different interactions. So, an efficient background learning method along with video summarization and generalized ROI is required with simultaneously tracks the interactions. Further, the issues with key-predictors has been discussed in section 5.1 and 5.2. respectively.

Issues
Resolution issues: From table 4, we can see that the resolution/ frame size of each dataset is different. The resolution directly affects ROI and processing time because a higher number of pixels take more time to process. Frame Rate issues: Also, different frame rate is a significant problem in processing. As a greater number of frames per second increase visibility but disproportionally affects the processing time in background generation methods. Interaction issues: From the samples of datasets it has been observed that in every dataset has different types of interaction i.e. a person-to-person, person-to-object. So, in order to get more optimal results to improve the AOD algorithm, an interaction factor needs to be considered. ROI issue: This is one of the major issues because ROI defines the area of processing from the complete frame to minimize computation. It has been observed that the ROI is statically defined and not work well in a few circumstances, for example, a video with higher resolution. So, a generalized method needs to be developed for ROI which is applicable to every dataset. Processing Time: The processing time evolved as another major issue because the usage of algorithm is insignificant if the processing will take more time. As, frame rate and resolution directly effecting it, so we need to work upon these two factors to reduce the processing time. Background Learning: By implementing the algorithm we also observed that the algorithm considers initial frames to learn the model for the background. But in a few cases like AVSS2007, it fails to learn. Also, in the case of recording the dataset, most of the capturing units are a fixed camera except in the case of VDAO. So, for improving the algorithm we assumed that training will be initiated after a few seconds of the recording.

Key-Predictors
The in-depth analysis of the issues results in various key predictors to improve the algorithm which are as follows: -CDR: It shows the model accuracy by considering all aspects of AOD. In the future, this measure is very helpful for validation of manual testing vs automated testing to enhance algorithm. gROI: It must be generalized as per the scene requirement such that the part of the scene which may have the area to put an object must be considered and it will be defined in the algorithm as per the resolution of capturing unit. Video Summarization: Instead of processing all frames, consider a few frames for processing. It can be accomplished by skipping the in-between frames of that second. Because we are more focused on the movements per second not in the movements within the seconds. Selective Background: For learning on background, consider the chosen frames through the video summarization process instead of working on all frames. Interaction factor: Identifying the level of interaction of a person towards the object and surroundings for more robustness in the algorithm. These factors are considered because it significantly improves the processing time and accuracy.

Conclusion
Identification of suspiciousness from an area under surveillance includes various activities and among them identifying an object is abandoned is challenging because no robust model found so far to identify abandoned object severity to cause damage. Therefore, in this paper authors have been tried to identify the challenges raised through testing the reproducibility of existing algorithms on the benchmark datasets. Therefore, in this paper authors observational parameters posed to define ground truths for dataset. Further, using various RQs, existing AOD method has been selected. The selected method's reproducibility tested on various benchmark dataset to identify the existing issues. Key-predictors has been identified using experimental analysis and observational parameters. In future, the researchers working in this area, may consider observational parameters for precise ground truth information. On the basis of the ground truths information collected through observational parameter, it can be used to with the key-predictors to develop a robust and automated computer vision-based algorithm for AOD.