Owing to effective and flexible data acquisition, unmanned aerial vehicle (UAV) has recently become a hotspot across the fields of computer vision (CV) and remote sensing (RS). Inspired by recent success of deep learning (DL), many advanced object detection and tracking approaches have been widely applied to various UAV-related tasks, such as environmental monitoring, precision agriculture, traffic management. This paper provides a comprehensive survey on the research progress and prospects of DL-based UAV object detection and tracking methods. More specifically, we first outline the challenges, statistics of existing methods, and provide solutions from the perspectives of DL-based models in three research topics: object detection from the image, object detection from the video, and object tracking from the video. Open datasets related to UAV-dominated object detection and tracking are exhausted, and four benchmark datasets are employed for performance evaluation using some state-of-the-art methods. Finally, prospects and considerations for the future work are discussed and summarized. It is expected that this survey can facilitate those researchers who come from remote sensing field with an overview of DL-based UAV object detection and tracking methods, along with some thoughts on their further developments.

Deep Learning for UAV-based Object Detection and Tracking: A Survey

To solve object detection issues in infrared images, such as a low recognition rate and a high false alarm rate caused by long distances, weak energy, and low resolution, we propose a region-free object detector named YOLO-FIR for infrared (IR) images with YOLOv5 core by compressing channels, optimizing parameters, etc. An improved infrared image object detection network, YOLO-FIRI, is further developed. Specifically, while designing the feature extraction network, the cross-stage-partial-connections (CSP) module in the shallow layer is expanded and iterated to maximize the use of shallow features. In addition, an improved attention module is introduced in residual blocks to focus on objects and suppress background. Moreover, multiscale detection is added to improve small object detection accuracy. Experimental results on the KAIST and FLIR datasets show that YOLO-FIRI demonstrates a qualitative improvement compared with the state-of-the-art detectors. Compared with YOLOv4, the mean average precision (mAP50) of YOLO-FIRI is increased by 21% on the KAIST dataset, the speed is reduced by 62%, the parameters are decreased by 89%, the weight size is reduced by more than 94%, and the computational costs are reduced by 84%. Compared with YOLO-FIR, YOLO-FIRI has an approximately 5% to 20% improvement in AP, AR (average recall), mAP50, F1, and mAP50:75. Furthermore, due to the shortcomings of high noise and weak features, image fusion can be applied to image preprocessing as a data enhancement method by fusing visible and infrared images based on a convolutional neural network.

YOLO-FIRI: Improved YOLOv5 for Infrared Image Object Detection

Analyzing videos and images captured by unmanned aerial vehicles or aerial drones is an emerging application attracting significant attention from researchers in various areas of computer vision. Currently, the major challenge is the development of autonomous operations to complete missions and replace human operators. In this paper, based on the type of analyzing videos and images captured by drones in computer vision, we have reviewed these applications by categorizing them into three groups. The first group is related to remote sensing with challenges such as camera calibration, image matching, and aerial triangulation. The second group is related to drone-autonomous navigation, in which computer vision methods are designed to explore challenges such as flight control, visual localization and mapping, and target tracking and obstacle detection. The third group is dedicated to using images and videos captured by drones in various applications, such as surveillance, agriculture and forestry, animal detection, disaster detection, and face recognition. Since most of the computer vision methods related to the three categories have been designed for real-world conditions, providing real conditions based on drones is impossible. We aim to explore papers that provide a database for these purposes. In the first two groups, some survey papers presented are current. However, the surveys have not been aimed at exploring any databases. This paper presents a complete review of databases in the first two groups and works that used the databases to apply their methods. Vision-based intelligent applications and their databases are explored in the third group, and we discuss open problems and avenues for future research.

Applications, databases and open computer vision research from drone videos and images: a survey

Owing to effective and flexible data acquisition, unmanned aerial vehicles (UAVs) have recently become a hotspot across the fields of computer vision (CV) and remote sensing (RS). Inspired by the recent success of deep learning (DL), many advanced object detection and tracking approaches have been widely applied to various UAV-related tasks, such as environmental monitoring, precision agriculture, and traffic management. 

/pdf/deep-learning-for-unmanned-aerial-vehicle-based-object-3ubewbcb.pdf

Deep Learning for Unmanned Aerial Vehicle-Based Object Detection and Tracking: A survey

The latest 5G mobile networks have enabled many exciting Internet of Things (IoT) applications that employ unmanned aerial vehicles (UAVs/drones). The success of most UAV-based IoT applications is heavily dependent on artificial intelligence (AI) technologies, for instance, computer vision and path planning. These AI methods must process data and provide decisions while ensuring low latency and low energy consumption. However, the existing cloud-based AI paradigm finds it difficult to meet these strict UAV requirements. Edge AI, which runs AI on-device or on edge servers close to users, can be suitable for improving UAV-based IoT services. This article provides a comprehensive analysis of the impact of edge AI on key UAV technical aspects (i.e., autonomous navigation, formation control, power management, security and privacy, computer vision, and communication) and applications (i.e., delivery systems, civil infrastructure inspection, precision agriculture, search and rescue (SAR) operations, acting as aerial wireless base stations (BSs), and drone light shows). As guidance for researchers and practitioners, this article also explores UAV-based edge AI implementation challenges, lessons learned, and future research directions.

https://ieeexplore.ieee.org/ielx7/6488907/6702522/09778241.pdf

A Survey on the Convergence of Edge Computing and AI for UAVs: Opportunities and Challenges

The spread of Unmanned Aerial Vehicles (UAVs) in the last decade revolutionized many applications fields. Most investigated research topics focus on increasing autonomy during operational campaigns, environmental monitoring, surveillance, maps, and labeling. To achieve such complex goals, a high-level module is exploited to build semantic knowledge leveraging the outputs of the low-level module that takes data acquired from multiple sensors and extracts information concerning what is sensed. All in all, the detection of the objects is undoubtedly the most important low-level task, and the most employed sensors to accomplish it are by far RGB cameras due to costs, dimensions, and the wide literature on RGB-based object detection. This survey presents recent advancements in 2D object detection for the case of UAVs, focusing on the differences, strategies, and trade-offs between the generic problem of object detection, and the adaptation of such solutions for operations of the UAV. Moreover, a new taxonomy that considers different heights intervals and driven by the methodological approaches introduced by the works in the state of the art instead of hardware, physical and/or technological constraints is proposed.

A Survey of Computer Vision Methods for 2D Object Detection from Unmanned Aerial Vehicles

Precise and robust localization is of fundamental importance for robots required to carry out autonomous tasks. Above all, in the case of Unmanned Aerial Vehicles (UAVs), efficiency and reliability are critical aspects in developing solutions for localization due to the limited computational capabilities, payload and power constraints. In this work, we leverage novel research in efficient deep neural architectures for the problem of 6 Degrees of Freedom (6-DoF) pose estimation from single RGB camera images. In particular, we introduce an efficient neural network to jointly regress the position and orientation of the camera with respect to the navigation environment. Experimental results show that the proposed network is capable of retaining similar results with respect to the most popular state of the art methods while being smaller and with lower latency, which are fundamental aspects for real-time robotics applications.

/pdf/faster-visual-based-localization-with-mobile-posenet-155jno41tm.pdf

Faster Visual-Based Localization with Mobile-PoseNet

The ability of the robots to imitate human movements has been an active research study since the dawn of the robotics. Obtaining a realistic imitation is essential in terms of perceived quality in human-robot interaction, but it is still a challenge due to the lack of effective mapping between human movements and the degrees of freedom of robotics systems. If high-level programming interfaces, software and simulation tools simplified robot programming, there is still a strong gap between robot control and natural user interfaces. In this paper, a system to reproduce on a robot the head movements of a user in the field of view of a consumer camera is presented. The system recognizes the presence of a user and its head pose in real-time by using a deep neural network, in order to extract head position angles and to command the robot head movements consequently, obtaining a realistic imitation. At the same time, the system represents a natural user interface to control the Aldebaran NAO and Pepper humanoid robots with the head movements, with applications in human-robot interaction.

/pdf/real-time-human-head-imitation-for-humanoid-robots-4610z8kvj7.pdf

Real-Time Human Head Imitation for Humanoid Robots

Efficient localisation plays a vital role in many modern applications of Unmanned Ground Vehicles (UGV) and Unmanned Aerial Vehicles (UAVs), which contributes to improved control, safety, power economy, etc. The ubiquitous 5G NR (New Radio) cellular network will provide new opportunities to enhance the localisation of UAVs and UGVs. In this paper, we review radio frequency (RF)-based approaches to localisation. We review the RF features that can be utilized for localisation and investigate the current methods suitable for Unmanned Vehicles under two general categories: range-based and fingerprinting. The existing state-of-the-art literature on RF-based localisation for both UAVs and UGVs is examined, and the envisioned 5G NR for localisation enhancement, and the future research direction are explored.

/pdf/a-review-of-radio-frequency-based-localisation-for-aerial-3jk4dzx5.pdf

A Review of Radio Frequency Based Localisation for Aerial and Ground Robots with 5G Future Perspectives

Robot self-localization is essential for operating autonomously in open environments. When cameras are the main source of information for retrieving the pose, numerous challenges are posed by the presence of dynamic objects, due to occlusion and continuous changes in the appearance. Recent research on global localization methods focused on using a single (or multiple) Convolutional Neural Network (CNN) to estimate the 6 Degrees of Freedom (6-DoF) pose directly from a monocular camera image. In contrast with the classical approaches using engineered feature detector, CNNs are usually more robust to environmental changes in light and to occlusions in outdoor scenarios. This paper contains an attempt to empirically demonstrate the ability of CNNs to ignore dynamic elements, such as pedestrians or cars, through learning. For this purpose, we pre-process a dataset for pose localization with an object segmentation network, masking potentially moving objects. Hence, we compare the pose regression CNN trained and/or tested on the set of masked images and the original one. Experimental results show that the performances of the two training approaches are similar, with a slight reduction of the error when hiding occluding objects from the views.

/pdf/a-case-study-on-the-impact-of-masking-moving-objects-on-the-1m5n9zthby.pdf

Claudio Cimarelli

Papers

A Survey of Computer Vision Methods for 2D Object Detection from Unmanned Aerial Vehicles

Faster Visual-Based Localization with Mobile-PoseNet

Real-Time Human Head Imitation for Humanoid Robots

A Review of Radio Frequency Based Localisation for Aerial and Ground Robots with 5G Future Perspectives

A case study on the impact of masking moving objects on the camera pose regression with CNNs