With growing numbers of intelligent autonomous systems in human environments, the ability of such systems to perceive, understand, and anticipate human behavior becomes increasingly important. Spec...

Human motion trajectory prediction: a survey:

With growing numbers of intelligent autonomous systems in human environments, the ability of such systems to perceive, understand and anticipate human behavior becomes increasingly important. Specifically, predicting future positions of dynamic agents and planning considering such predictions are key tasks for self-driving vehicles, service robots and advanced surveillance systems. This paper provides a survey of human motion trajectory prediction. We review, analyze and structure a large selection of work from different communities and propose a taxonomy that categorizes existing methods based on the motion modeling approach and level of contextual information used. We provide an overview of the existing datasets and performance metrics. We discuss limitations of the state of the art and outline directions for further research.

Human Motion Trajectory Prediction: A Survey

How should representations from complementary sensors be integrated for autonomous driving? Geometry-based sensor fusion has shown great promise for perception tasks such as object detection and motion forecasting. However, for the actual driving task, the global context of the 3D scene is key, e.g. a change in traffic light state can affect the behavior of a vehicle geometrically distant from that traffic light. Geometry alone may therefore be insufficient for effectively fusing representations in end-to-end driving models. In this work, we demonstrate that imitation learning policies based on existing sensor fusion methods under-perform in the presence of a high density of dynamic agents and complex scenarios, which require global contextual reasoning, such as handling traffic oncoming from multiple directions at uncontrolled intersections. Therefore, we propose TransFuser, a novel Multi-Modal Fusion Transformer, to integrate image and LiDAR representations using attention. We experimentally validate the efficacy of our approach in urban settings involving complex scenarios using the CARLA urban driving simulator. Our approach achieves state-of-the-art driving performance while reducing collisions by 76% compared to geometry-based fusion.

/pdf/multi-modal-fusion-transformer-for-end-to-end-autonomous-4x0ffkhxd5.pdf

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

For next-generation smart cities, small UAVs (also known as drones) are vital to incorporate in airspace for advancing the transportation systems. This paper presents a review of recent developments in relation to the application of UAVs in three major domains of transportation, namely; road safety, traffic monitoring and highway infrastructure management. Advances in computer vision algorithms to extract key features from UAV acquired videos and images are discussed along with the discussion on improvements made in traffic flow analysis methods, risk assessment and assistance in accident investigation and damage assessments for bridges and pavements. Additionally, barriers associated with the wide-scale deployment of UAVs technology are identified and countermeasures to overcome these barriers are discussed, along with their implications.

Applications of unmanned aerial vehicle (UAV) in road safety, traffic and highway infrastructure management: Recent advances and challenges.

Understanding crowd motion dynamics is critical to real-world applications, e.g., surveillance systems and autonomous driving. This is challenging because it requires effectively modeling the socially aware crowd spatial interaction and complex temporal dependencies. We believe attention is the most important factor for trajectory prediction. In this paper, we present STAR, a Spatio-Temporal grAph tRansformer framework, which tackles trajectory prediction by only attention mechanisms. STAR models intra-graph crowd interaction by TGConv, a novel Transformer-based graph convolution mechanism. The inter-graph temporal dependencies are modeled by separate temporal Transformers. STAR captures complex spatio-temporal interactions by interleaving between spatial and temporal Transformers. To calibrate the temporal prediction for the long-lasting effect of disappeared pedestrians, we introduce a read-writable external memory module, consistently being updated by the temporal Transformer. We show that with only attention mechanism, STAR achieves state-of-the-art performance on 5 commonly used real-world pedestrian prediction datasets.

Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction

Human trajectory prediction is challenging and critical in various applications (e.g., autonomous vehicles and social robots). Because of the continuity and foresight of the pedestrian movements, the moving pedestrians in crowded spaces will consider both spatial and temporal interactions to avoid future collisions. However, most of the existing methods ignore the temporal correlations of interactions with other pedestrians involved in a scene. In this work, we propose a Spatial-Temporal Graph Attention network (STGAT), based on a sequence-to-sequence architecture to predict future trajectories of pedestrians. Besides the spatial interactions captured by the graph attention mechanism at each time-step, we adopt an extra LSTM to encode the temporal correlations of interactions. Through comparisons with state-of-the-art methods, our model achieves superior performance on two publicly available crowd datasets (ETH and UCY) and produces more "socially" plausible trajectories for pedestrians.

/pdf/stgat-modeling-spatial-temporal-interactions-for-human-4rfhczp3fm.pdf

STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction

School of Electronic Engineering, University of Beijing Posts and Telecommunications, Beijing 100876, China; Institute of Electronic and Information Engineering in Guangdong, University of Electronic Science and Technology of China, Dongguan 523808, China; School of Control and Computer Engineering, North China Electric Power University, Beijing 102206, China; Beijing Key Lab of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China

Vehicle tracking by detection in UAV aerial video

Monocular depth estimation is a challenging problem in computer vision and is crucial for understanding 3D scene geometry. Recently, deep convolutional neural networks (DCNNs) based methods have improved the estimation accuracy significantly. However, existing methods fail to consider complex textures and geometries in scenes, thereby resulting in loss of local details, distorted object boundaries, and blurry reconstruction. In this paper, we proposed an end-to-end multi-scale residual pyramid attention network (MRPAN) to mitigate these problems. First, we propose a multi-scale attention context aggregation (MACA) module, which consists of spatial attention module (SAM) and global attention module (GAM). By considering the position and scale correlation of pixels from spatial and global perspectives, the proposed module can adaptively learn the similarity between pixels so as to obtain more global context information of the image and recover complex structures in the scene. Then we proposed an improved residual refinement module (RRM) to further refine the scene structure, giving rise to deeper semantic information and retain more local details. Experimental results show that our method achieves more promising performance in object boundaries and local details compared with other state-of-the-art methods.

Multi-Scale Residual Pyramid Attention Network for Monocular Depth Estimation

Image-based rendering (IBR) attempts to synthesize novel views using a set of observed images Some IBR approaches (such as light fields) have yielded impressive high-quality results on small-scale scenes with dense photo capture However, available wide-baseline IBR methods are still restricted by the low geometric accuracy and completeness of multi-view stereo (MVS) reconstruction on low-textured and non-Lambertian surfaces The issues become more significant in large-scale outdoor scenes due to challenging scene content, eg, buildings, trees, and sky To address these problems, we present a novel IBR algorithm that consists of two key components First, we propose a novel depth refinement method that combines MVS depth maps with monocular depth maps predicted via deep learning A lookup table remap is proposed for converting the scale of the monocular depths to be consistent with the scale of the MVS depths Then, the rescaled monocular depth is used as the constraint in the minimum spanning tree (MST)-based nonlocal filter to refine the per-view MVS depth Second, we present an efficient shape-preserving warping algorithm that uses superpixels to generate the warped images and blend expected novel views of scenes The proposed method has been evaluated on public MVS and view synthesis datasets, as well as newly captured large-scale outdoor datasets In comparison with state-of-the-art methods, the experimental results demonstrated that the proposed method can obtain more complete and reliable depth maps for the challenging large-scale outdoor scenes, thereby resulting in more promising novel view synthesis

/pdf/image-based-rendering-for-large-scale-outdoor-scenes-with-329do9dbj4.pdf

Zhaoxin Li

Papers

STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction

Vehicle tracking by detection in UAV aerial video

Multi-Scale Residual Pyramid Attention Network for Monocular Depth Estimation

Image-Based Rendering for Large-Scale Outdoor Scenes With Fusion of Monocular and Multi-View Stereo Depth