With the rapid development of convolutional neural networks (CNNs), significant progress has been achieved in semantic segmentation. Despite the great success, such deep learning approaches require large scale real-world datasets with pixel-level annotations. However, considering that pixel-level labeling of semantics is extremely laborious, many researchers turn to utilize synthetic data with free annotations. But due to the clear domain gap, the segmentation model trained with the synthetic images tends to perform poorly on the real-world datasets. Unsupervised domain adaptation (UDA) for semantic segmentation recently gains an increasing research attention, which aims at alleviating the domain discrepancy. Existing methods in this scope either simply align features or the outputs across the source and target domains or have to deal with the complex image processing and post-processing problems. In this work, we propose a novel multi-level UDA model named Confidence-and-Refinement Adaptation Model (CRAM), which contains a confidence-aware entropy alignment (CEA) module and a style feature alignment (SFA) module. Through CEA, the adaptation is done locally via adversarial learning in the output space, making the segmentation model pay attention to the high-confident predictions. Furthermore, to enhance the model transfer in the shallow feature space, the SFA module is applied to minimize the appearance gap across domains. Experiments on two challenging UDA benchmarks “GTA5-to-Cityscapes” and “SYNTHIA-to-Cityscapes” demonstrate the effectiveness of CRAM. We achieve comparable performance with the existing state-of-the-art works with advantages in simplicity and convergence speed.

/pdf/confidence-and-refinement-adaptation-model-for-cross-domain-3qoid2f0.pdf

Confidence-and-Refinement Adaptation Model for Cross-Domain Semantic Segmentation

Semantic segmentation and depth estimation lie at the heart of scene understanding and play crucial roles especially for autonomous driving. In particular, it is desirable for an intelligent self-driving agent to discern unexpected obstacles on the road ahead reliably in real-time. While existing semantic segmentation studies for small road hazard detection have incorporated fusion of multiple modalities, they require additional sensor inputs and are often limited by a heavyweight network for real-time processing. In this light, we propose an end-to-end Real-time Obstacle Detection via Simultaneous refinement, coined RODSNet (<uri>https://github.com/SAMMiCA/</uri> RODSNet) which jointly learns semantic segmentation and disparity maps from a stereo RGB pair and refines them simultaneously in a single module. RODSNet exploits two efficient single-task network architectures and a simple refinement module in a multi-task learning scheme to recognize unexpected small obstacles on the road. We validate our method by fusing <italic>Cityscapes</italic> and <italic>Lost and Found</italic> datasets and show that our method outperforms previous approaches on the obstacle detection task, even recognizing the unannotated obstacles at 14.5 FPS on our fused dataset (<inline-formula> <tex-math notation="LaTeX">$2048\times 1024$ </tex-math></inline-formula> resolution) using RODSNet<monospace>-<inline-formula> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula></monospace>. In addition, extensive ablation studies demonstrate that our simultaneous refinement effectively facilitates contextual learning between semantic and depth information.

End-to-End Real-Time Obstacle Detection Network for Safe Self-Driving via Multi-Task Learning

Semantic segmentation has attracted increasing attention due to its important role in self-driving, and it is often realized by supervised learning with large number of well labeled maps. However, the labeled images are hard to be obtained in most circumstances, and the common way for unsupervised semantic segmentation is usually implemented by transferring the knowledge from source supervised domain to target unsupervised domain. Most researches focus on encouraging target predictions to be closer to the source ones through a weight-sharing network, and achieve certain performance. However, these methods often suffer from the domain shift problem that the networks are often trained towards the source domain and lead to performance degradation. In this paper, we propose a target-targeted domain adaptation approach by focusing the training on target domain. Our model consists of two components: the Image-to-image Translation (IIT) module to translate the source image to target domain and the Target-targeted Segmentation Adaptation (TSA) module to focus the semantic segmentation on target domain. The IIT module deals with image space alignment while the TSA module bridges the domain gap at the segmentation map level. In addition, we design a closed-loop learning to promote each other by employing feedback from TSA to IIT. Extensive experiments on GTA5 and SYNTHIA to Cityscapes demonstrate the effectiveness of our method in domain adaptation of unsupervised semantic segmentation.

Target-targeted Domain Adaptation for Unsupervised Semantic Segmentation

Semantic segmentation and depth estimation lie at the heart of scene understanding and play crucial roles especially for autonomous driving. In particular, it is desirable for an intelligent self-driving agent to discern unexpected obstacles on the road ahead reliably in real-time. While existing semantic segmentation studies for small road hazard detection have incorporated fusion of multiple modalities, they require additional sensor inputs and are often limited by a heavyweight network for real-time processing. In this light, we propose an end-to-end Real-time Obstacle Detection via Simultaneous refinement, coined RODSNet ( <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/SAMMiCA/</uri> RODSNet) which jointly learns semantic segmentation and disparity maps from a stereo RGB pair and refines them simultaneously in a single module. RODSNet exploits two efficient single-task network architectures and a simple refinement module in a multi-task learning scheme to recognize unexpected small obstacles on the road. We validate our method by fusing <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cityscapes</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lost and Found</i> datasets and show that our method outperforms previous approaches on the obstacle detection task, even recognizing the unannotated obstacles at 14.5 FPS on our fused dataset ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2048\times 1024$ </tex-math></inline-formula> resolution) using RODSNet <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-<inline-formula> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula></monospace> . In addition, extensive ablation studies demonstrate that our simultaneous refinement effectively facilitates contextual learning between semantic and depth information. 

Deconvolutional networks have attracted extensive attention and have been successfully applied in the field of computer vision. In this paper we propose a novel two-branch deconvolutional network (TBDN) that can improve the performance of conventional deconvolutional networks and reduce the computational complexity. A feasible iterative algorithm is designed to solve the optimization problem for the TBDN model, and a theoretical analysis of the convergence and computational complexity for the algorithm is also provided. The application of the TBDN in stereo matching is presented by constructing a disparity estimation network. Extensive experimental results on four commonly used datasets demonstrate the efficiency and effectiveness of the proposed TBDN. 

Two-Branch Deconvolutional Network With Application in Stereo Matching

Semantic segmentation and depth estimation play important roles in the field of autonomous driving. In recent years, the advantages of Convolutional Neural Networks (CNNs) have allowed these two topics to flourish. However, people always solve these two tasks separately and rarely solve them in a united model. In this paper, we propose a Mutual Encouragement Network (MENet), which includes a semantic segmentation branch and a disparity regression branch, and simultaneously generates semantic map and visual disparity. In the cost volume construction phase, the depth information is embedded in the semantic segmentation branch to increase contextual understanding. Similarly, the semantic information is also included in the disparity regression branch to generate more accurate disparity. Two branches mutually promote each other during training phase and inference phase. We conducted our method on the popular dataset KITTI, and the experimental results show that our method can outperform the state-of-the-art methods on both visual disparity generation and semantic segmentation. In addition, extensive ablation studies also demonstrate that the two tasks in our method can facilitate each other significantly with the proposed approach.

When Visual Disparity Generation Meets Semantic Segmentation: A Mutual Encouragement Approach

Object detection, as one of the most important environment perception tasks for traffic safety in intelligent transportation systems, has been widely investigated recently. However, most of the researches focus on the fully supervised scenario, and inevitably lead to model failure. With the continuous development of Zero-Shot Learning (ZSL) models, Generalized Zero-Shot Detection (GZSD) has attracted great attention due to its ability of detecting unseen objects. Many researchers tend to map the detected visual features to semantic attributes and then separate seen and unseen domains during inference. But they have ignore that the generative methods generally have higher performance than these visual-semantic mapping methods, and they have been confirmed from previous GZSL methods. In order to make up for the vacancy of GZSD in the generative methods, we propose an idea of using curriculum learning to generate more precise unseen visual features. And with the excellent performance of WGAN-based method in sample synthesis, we realize the function of using semantics to generate visual features for unseen domains. In addition, we also adopt part of the idea of meta-learning to progressively correct the capability of the generator for better mitigating domain shift problem during the generation process. Through the above ideas, we can detect both seen and unseen bounding boxes and classify them accurately, by combining with the excellent detection ability of Faster-RCNN. Extensive experimental results on two popular datasets, i.e., MSCOCO and KITTI, show that our proposed method can outperform the state-of-the-art methods.

Yi Chen

Papers

When Visual Disparity Generation Meets Semantic Segmentation: A Mutual Encouragement Approach

Confidence-and-Refinement Adaptation Model for Cross-Domain Semantic Segmentation

From Less to More: Progressive Generalized Zero-Shot Detection With Curriculum Learning