Detection of Potholes on Roads using a Drone

Locating potholes and repairing them is essential, but it has always been a time consuming task for the authorities. This paper presents a way that can help the authorities speed up the pothole detection process by the use of a camera-enabled Unmanned Aerial Vehicle drone. The system is further enabled with a geo-tag and reports the presence of a pothole to the central database which is accessible by the relevant authorities and the common road users. The potholes are located on an open-source map, through which the users using the road can take caution. This increases public safety and helps the concerned authorities take action faster. The model is trained with YOLOv3 algorithm to even detect potholes filled with water, and distinguish potholes from dark road patches, and etc. The results show good accuracy of 85% in detecting the potholes with a low false-negative and false-positive rate.


Introduction
Roads are an essential means of transportation. It carries a major percentage of passenger traffic of any country. It is known that most of the roads in developing countries would be congested and narrow resulting in poor surface quality and the maintenance of the roads is not satisfactory. Due to poor maintenance, poor surface quality or due to a variety of other reasons a pothole could be formed on roads. A pothole is a kind of disruption in the surface of a road in which a portion of the road material is damaged. In rainy seasons, if these potholes are covered by water, it might lead to accidents which might be fatal too. It gets difficult to track all the potholes by the concerned authorities.
The aim of this paper is to develop a "Pothole Detection System", which assists the concerned authorities to find and fix the potholes. A manually controlled or automatic drone will be equipped with the Pothole detection system which uses technologies such as Machine Learning and Computer vision to identify the potholes on roads.
In today's day and age Unmanned Aerial Vehicles (UAVs), also called drones have a wide variety of applications. They are used for military purposes, they can be * Corresponding author. Email: Bhhemaraj@gmail.com used to predict the crop quality, they are being used in photography to capture stunning pictures and videos of landscapes. Also, in the recent times various large organizations are trying to deliver goods through drones. The utility of drones is boundless, its applications can be seen in every industry and vertical.
Artificial intelligence also has a wide variety of applications in various fields such as finance, automotive, marketing, health care and etc. Almost every organization today has adopted AI in one way or the other.
In this work, an effort is being made to identify the potholes and report them to the relevant authorities which can help them take action. The project uses a camera as a sensor to scan the roads and detect potholes. Images and videos are captured in real-time and the YOLOv3 image processing algorithm is used. The project also uses an array of depthsensing systems to detect, geo-tag, and report the presence of a pothole to a centralized database, which is further sent to the concerned authorities of the roads in the particular area, it can also be accessed by the common public through an opensource map which can help them take caution while traveling on roads with potholes, this helps in enhancing the public safety and enables the concerned authorities to fix the potholes faster.

Related Work
Vigneshwar K et. al. [1], have classified potholes using various techniques like Image filtering, Image segmentation and clustering techniques like edge detection, K-Means, thresholding and Fuzzy C-Means. This paper focuses mainly on comparing various image segmentation techniques rather than building a standalone product. Shambhu Hegde et. al. [2], have the vehicle fitted with an ultrasonic sensor used to measure the potholes depth. The output of this sensor is consumed and processed by a microcontroller which relays the information to other vehicles. The detection of potholes is through computed values of the sensor and no particular classification mechanism is used.
VinayRishiwal et.al. [3] used the accelerometer which is built-in an android device. Then collected the axis acceleration data built and used it. Potholes are detected by monitoring the z-axis acceleration which undergoes sudden changes when a pothole is encountered.
Kiran Kumar et.al. [4], have fitted the vehicle with a laser and a camera set up and the picture of the reflected laser is taken by the camera and is used to detect potholes. It may have poor accuracy as it is a physics based approach and involves complex mathematical calculations.
Supervised learning based approach is followed for identification of images with potholes and without potholes by Kanza Azhar et. al. [5]. HOG Feature descriptors are used for detection and a naïve-Bayes classifier is used to classify the image as a pothole or not. The main drawback could be with the naïve-Bayes classifier which could have a poor accuracy.
Vosco Pereira et. al. [6], has the data gathered by manual clicking of photos and also through datasets available on the internet. A model based on convolutional neural networks is used to perform the classification. The limitation is that this model does not account for overfitting caused in the Deep Learning model.
For Detection of Pothole and Warning System, Sudarshan S Rode et. al. [7], have used an architecture based on Wi-Fi. The system helps the driver of the vehicle to escape the potholes on the roads by providing warnings earlier. The pothole detection recommended in this paper facilitates the driver of the vehicle to obtain statistics of the potholes on the roads in the surrounding area of the vehicle which is moving. The detection system could be used along with the vehicle to alarm the driver of the vehicle by using a visual or audio signal or sometime it might activate the braking system.
Byeong -ho Kang et. al. [8], developed a system for detecting potholes by making use of 2D LiDAR and a camera to improvise the accuracy of the identification of a pothole system. The information of distance and angle of the road are acquired by using a 2DLiDAR. It includes reduction of noise, clustering, extraction of line segments, pre-processing and gradient of pothole data function. The pothole detection using image-based technique is applied to enhance the accuracy of the detection of pothole and to get the shape of the pothole. This comprises of filtering of noise, addictive noise filtering, binarization, edge extraction, brightness control, object extraction and detection of pothole.
Ionut Schiopu et. al. [9], proposes a technique for discovery and stalking of potholes in videos shot by the camera positioned inside a moving car. The detection of the potholes is done through the image area wherever the road is seen with the highest resolution. A set of candidate regions are generated by an algorithm. The features like its size, intensity surface regularity, the background model, the length of the contour and also the shape of the pothole region are extracted. Based on these features, a decision tree labels the candidate regions as putative potholes. This eliminates the false positives because of the shadows of other objects. The presumed potholes which are tracked effectively in consecutive frames are lastly confirmed as potholes.
Higher accuracy was achieved with a low false rate by Sung Won Lee et. al. [10]. The author compared six of the image processing algorithms, like convex hull, background subtractor, wavelet energy field, saliency map the optimized combination. The Otsu binary algorithm was used to find the threshold value within 0 to 255. Background subtractor produces a background image then subtracts the overlays and divides the object using the above 7 algorithms to detect potholes within images and compare results of 6 most accurate detections to detect potholes. The system had an accuracy of 90%.
Amita et. al. [11], developed 2 methods constructed on stereo-vision and deep learning. A. Single-Frame Stereo Vision-Based Technique which detect potholes detecting areas on the road evaluated as 'below road surface' by using 1) Line Fitting, 2) 3D Plane Fitting B. Multi-Frame Fusion-Based Technique The approach named Digital elevation model -a multi-frame fusion for better digital elevation models. Anup graded approach from the single-frame method uses 1) Digital Elevation Model, 2) Visual Odometry. The main modules within Mask R-CNN are: 1) Region Proposal Network, 2) Refined Bounding Boxes, 3) Instance Masks Overall precision and recall for 50 random frames from our test dataset is 87%. Amila Akagic et. al. [12], used a method that removes asphalt by analyzing colour later by carrying out image segmentation. After detection of asphalt, exploration continues within the spotted region. The methodology has 3 steps: Pre-processing-This step has two stages namely: resizing and segmentation. Region Extraction-The region is the area where potholes are found. Pothole detection-This last step has three phases: cropped image and Otsu results. The testing was done on 80+ pot-hole images captured from several cameras, in different angles with irregular shapes and numerous potholes with varied sizes. Average execution time is 465.71ms with average error of 7%. Recognition is successful if it discovered more than 50% of potholes in the image and had an accuracy of82%.
Hanan Elazhary [13] explains how the world is undergoing a technological shift and how combining internet with things can be beneficial and important The following are the short-comings and research gaps in existing work:

EAI Endorsed Transactions on
Energy Web 11 2021 -05 2022 | Volume 9 | Issue 38 | e4 Detection of Potholes on Roads using a Drone 3 The current system in place to detect and report potholes is very slow to reach concerned authorities and opaque in its functioning. It is very inefficient in its working without any assurance that the complaint registered about a pothole will ever see any action against it. Even with a few technology solutions out there the sustainability is a huge question mark due to its inaccuracy and hassle to operate effectively.
Based on the literature survey, it is safe to conclude that the objective of detecting potholes could be achieved in a variety of ways. However, since the application involves maintaining a database of potholes along with their geographic locations, there are a lot of gaps that need to be addressed.
One of the challenges is to differentiate potholes from patches. Though both might look similar, building a model that could differentiate the two is an important fact to consider.
Detecting potholes with high accuracy is the need of the hour as all applications require high accuracy to function correctly. Communication and interactions with the concerned government authorities is the other bigger challenge.

Methodology
The proposed work intends to make a Pothole Detection System using a drone. It assists the concerned authorities to find and fix the issue. A manually or an automatically controlled drone will be equipped with the Pothole detection system. It uses a 'Camera' as sensor to scan the road for any potholes.
The camera then captures the images / video in real time. The system uses technologies such as Deep Learning to achieve the results. It then maps the detected pothole on a map application. The data is pushed to a government server on a daily basis. It also alerts the map users to potential potholes on route.7.

System Architecture
As seen from the architecture diagram the system has a drone which is central to data and footage gathering using a high resolution camera system attached to the drone, every single latitude or longitude change is recorded and time stamped to determine the exact time and location of any detected potholes using the GPS module.

Figure 1. System Architecture
Using the wireless module, the drone communicates with a laptop for control instructions while also relaying data received from its sensors. This data is processed using the pothole detection algorithms and all relevant data is stored onto a centralized database. This data base houses all aspects of detected pothole related information such as when and where the pothole was detected and also its severity.
This database is updated with relevant government authorities in real time to help maintain our roads better. The precise location of the pothole is also marked on an open source maps. The precise location of the pothole is also marked on an open source maps giving the general public access to live location of potholes and hence helping then travel safer and more comfortably.
A drone equipped with cameras is used to scan the roads by flying over them. This feed is then relayed to the remote system for processing. The drone uses a high resolution camera to record and stream the video to a laptop or a mobile device. This feed is used for subsequent processing and detection of potholes and its mapping. In the present work, YOLOv3 is used to distinguish potholes. YOLO uses a completely unique strategy. It is anything but a customary classifier that is repurposed to be an item finder. YOLO scans the picture only once (henceforth its name: You Only Look Once) however in a smart way. It divides the image into a grid size 13 by 13 cells:

Figure 3. YOLOv3
Each of the cells is responsible for looking into 5 bound boxes. Items are depicted by a square shaped box. YOLO generates a certainty score that reveals to us how sure it is that the jumping box really represents some item. This score doesn't identify the type of object within the container. Image 4 shown below depicts objects encased within rectangle shaped jumping boxes (the higher the certainty score, the bigger the crate is drawn): For each bound box, the cells additionally denote a class. This works like a classifier: it denotes a likelihood conveyance over all the potential classes. The variant of YOLO utilized in this work is prepared on the PASCALVOC dataset, which can recognize over 20 different classes, example, boat, pizza, drone, car, cat, bicycle and dog.
The value of the certainty score for the bound box and the class forecast are combined into a final score that reveals to us the likelihood that the bound box represents a particular kind of item. For example, the large yellow box on the left as shown in the below image is 85% certain that it contains the article "dog":  As there are 13×13 = 169 matrix cells and every cell predicts 5 bound boxes, we end up with 845 jumping encloses complete. Incidentally, the majority of these containers will have lower certainty scores, so we can just take up the cases that have a final score of 30% or more (The edge contingent can be changes upon how exact the indicator needs to be). The final prediction is then:

Figure 6. YOLOv3 detection
From the 845 all out bouncing boxes just three boxes were kept as they denoted the highest outcomes. Yet, note that inspite of the fact that there were 845 separate expectations, they were completely made simultaneously -the neural network just ran once. Also, that is the reason YOLO is incredible and quick.

Results and Discussions
Pothole detection on images: In this module, an image is uploaded and the corresponding area is entered. After clicking on submit, the system performs object detection using a well-trained YOLO V3 model to detect potholes in the image.
Once the image is captured and transferred to the system from the drone, image is uploaded in this menu to detect potholes.
The object detection model processes the image and marks the potholes in the image with bounding boxes along with the accuracy or confidence levels with which the model predicts or detects the potholes.

EAI Endorsed Transactions on
Energy Web 11 2021 -05 2022 | Volume 9 | Issue 38 | e4 Again, the output of the system is an image with the potholes marked. This image is converted into binary form and is then stored in the database against the location entered.
The Model is able to predict the potholes with an accuracy close to 85% and perform as well even on potholes filled with water. Figure 7. Displays the input image uploaded to the system and Figure 8 is the output image produced by the system.
As displayed, the potholes are marked in green coloured boxes along with the confidence levels with which these potholes are detected by the system.   Once the view button is clicked, the database is queried with the area/location entered and the image corresponding to area is retrieved, converted to human interpretable form and then displayed onto the screen.
This Tab allows authorities to check all the potholes at a particular location and plan their resources appropriately to cater to it.

Conclusion
It is believed that a pothole detection and mapping system is essential to not only to prevent road-based accidents but to also maintain better quality of roads within our country which contributes to a faster development. It is believed that a pothole detector and mapping system will help save lives, make a road user experience more enjoyable while also making it easier for governments to maintain the most crucial transportation system in the country.
The system accurately detects potholes on a road through imaging and creates an open source map-based database of potholes on the road that is updated in real time. It also provides real time pothole information to the government and general public through the open source map which enable the government to maintain and fix potholes faster. The system developed has an accuracy of 85% and also has a low rate of false-positives and falsenegatives.