When building a unified vision system or gradually adding new capabilities to a system, the usual assumption is that training data for all tasks is always available. However, as the number of tasks grows, storing and retraining on such data becomes infeasible. A new problem arises where we add new capabilities to a Convolutional Neural Network (CNN), but the training data for its existing capabilities are unavailable. We propose our Learning without Forgetting method, which uses only new task data to train the network while preserving the original capabilities. Our method performs favorably compared to commonly used feature extraction and fine-tuning adaption techniques and performs similarly to multitask learning that uses original task data we assume unavailable. A more surprising observation is that Learning without Forgetting may be able to replace fine-tuning with similar old and new task datasets for improved new task performance.

Learning without Forgetting

Deep learning in environmental remote sensing: Achievements and challenges

With the development of satellite technology, up to date imaging mode of synthetic aperture radar (SAR) satellite can provide higher resolution SAR imageries, which benefits ship detection and instance segmentation. Meanwhile, object detectors based on convolutional neural network (CNN) show high performance on SAR ship detection even without land-ocean segmentation; but with respective shortcomings, such as the relatively small size of SAR images for ship detection, limited SAR training samples, and inappropriate annotations, in existing SAR ship datasets, related research is hampered. To promote the development of CNN based ship detection and instance segmentation, we have constructed a High-Resolution SAR Images Dataset (HRSID). In addition to object detection, instance segmentation can also be implemented on HRSID. As for dataset construction, under the overlapped ratio of 25%, 136 panoramic SAR imageries with ranging resolution from 1m to 5m are cropped to $800 \times 800$ pixels SAR images. To reduce wrong annotation and missing annotation, optical remote sensing imageries are applied to reduce the interferes from harbor constructions. There are 5604 cropped SAR images and 16951 ships in HRSID, and we have divided HRSID into a training set (65% SAR images) and test set (35% SAR images) with the format of Microsoft Common Objects in Context (MS COCO). 8 state-of-the-art detectors are experimented on HRSID to build the baseline; MS COCO evaluation metrics are applicated for comprehensive evaluation. Experimental results reveal that ship detection and instance segmentation can be well implemented on HRSID.

/pdf/hrsid-a-high-resolution-sar-images-dataset-for-ship-1mwaxlmjdm.pdf

HRSID: A High-Resolution SAR Images Dataset for Ship Detection and Instance Segmentation

With the development of deep learning (DL) and synthetic aperture radar (SAR) imaging techniques, SAR automatic target recognition has come to a breakthrough. Numerous algorithms have been proposed and competitive results have been achieved in detecting different targets. However, due to the influence of various sizes and complex background of ships, detecting multiscale ships in SAR images is still challenging. To solve the problems, a novel network, called attention receptive pyramid network (ARPN), is proposed in this article. ARPN is a two-stage detector and designed to improve the performance of detecting multiscale ships in SAR images by enhancing the relationships among nonlocal features and refining information at different feature maps. Specifically, receptive fields block (RFB) and convolutional block attention module (CBAM) are employed and combined reasonably in attention receptive block to build a top-down fine-grained feature pyramid. RFB, composed of several branches of convolutional layers with specifically asymmetric kernel sizes and various dilation rates, is used for grabbing features of ships with large aspect ratios and enhancing local features with their global dependences. CBAM, which consists of channel and spatial attention mechanisms, is utilized to boost significant information and suppress interference caused by surroundings. To evaluate the effectiveness of ARPN, experiments are conducted on SAR Ship Detection Dataset and two large-scene SAR images. The detection results illustrate that competitive performance has been achieved by our method in comparison with several CNN-based algorithms, e.g., Faster-RCNN, RetinaNet, feature pyramid network, YOLOv3, Dense Attention Pyramid Network, Depth-wise Separable Convolutional Neural Network, High-Resolution Ship Detection Network, and Squeeze and Excitation Rank Faster-RCNN.

/pdf/attention-receptive-pyramid-network-for-ship-detection-in-4h5icl2rn3.pdf

Attention Receptive Pyramid Network for Ship Detection in SAR Images

Polarisation Applications In Remote Sensing

Synthetic aperture radar (SAR) is an active microwave imaging sensor with the capability of working in all-weather, all-day to provide high-resolution SAR images. Recently, SAR images have been widely used in civilian and military fields, such as ship detection. The scales of different ships vary in SAR images, especially for small-scale ships, which only occupy few pixels and have lower contrast. Compared with large-scale ships, the current ship detection methods are insensitive to small-scale ships. Therefore, the ship detection methods are facing difficulties with multi-scale ship detection in SAR images. A novel multi-scale ship detection method based on a dense attention pyramid network (DAPN) in SAR images is proposed in this paper. The DAPN adopts a pyramid structure, which densely connects convolutional block attention module (CBAM) to each concatenated feature map from top to bottom of the pyramid network. In this way, abundant features containing resolution and semantic information are extracted for multi-scale ship detection while refining concatenated feature maps to highlight salient features for specific scales by CBAM. Then, the salient features are integrated with global unblurred features to improve accuracy effectively in SAR images. Finally, the fused feature maps are fed to the detection network to obtain the final detection results. Experiments on the data set of SAR ship detection data set (SSDD) including multi-scale ships in various SAR images show that the proposed method can detect multi-scale ships in different scenes of SAR images with extremely high accuracy and outperforms other ship detection methods implemented on SSDD.

Dense Attention Pyramid Networks for Multi-Scale Ship Detection in SAR Images

As a mission-critical sensor, SAR has been applied in environmental monitoring and battlefield surveillance; moreover, SAR target recognition is one of the most important applications of SAR technology. However, in practical applications, the number of samples available for training is relatively small, so the SAR target recognition can be regarded as a small sample recognition problem. One of the main directions to solve the small sample recognition problem is to realize the data augmentation. Therefore, a SAR image data augmentation method via Generative Adversarial Nets (GAN) is proposed in this paper. The method uses Wasserstein GAN with a gradient penalty (WGAN-GP) to generate new samples based on existing SAR data, which can augment the sample number in training dataset. Meanwhile, the sample selection filters are designed to extract the generated samples with high quality and specific azimuth, which can avoid the randomness of the data augmentation, and improve the quality of the newly generated training samples. The experiments based on MSTAR data show that, for three-class recognition problem, when the training sample is only 108, the proposed method can improve the recognition rate from 79% to 91.6%; and for ten-class recognition problem, when the training sample is only 360, the proposed method can improve the recognition rate from 57.48% to 79.59%. Compared with the traditional data linear generation method, the proposed method shows significant improvement on the quantity and quality of the training samples, and can effectively solve the problem of the small sample recognition.

/pdf/image-data-augmentation-for-sar-sensor-via-generative-2fg53egu1o.pdf

Image Data Augmentation for SAR Sensor via Generative Adversarial Nets

Incremental learning methods update the existing model with new knowledge when the target data increase continuously. Open set recognition (OSR) algorithms provide classifiers with a rejection option so that the new untrained target type is identified. In this paper, an open set incremental learning method is introduced for automatic target recognition, which is able to recognize and learn the new unknown classes continually. The proposed method, open set model with incremental learning (OSmIL), is an ensemble classifier so it is able to be updated only by the new data. For saving the computational time and storage source, a new exemplar selection method is introduced for model simplifying. Edge samples are selected to cover training classes; as a result, the model size is deduced and controlled. Moreover, because extreme value theory (EVT) is suitable to fit a classification model that includes open space risk, the decision function based on EVT makes an open set classifier for identifying the new classes. Experimental results demonstrate that the proposed OSmIL outperforms the other state of the arts on the accuracy of multiclass OSR. And OSmIL can maintain good accuracy and efficiency in the incremental learning experiment set.

Open Set Incremental Learning for Automatic Target Recognition

Convolutional neural networks (CNNs) have been widely used in synthetic aperture radar (SAR) target recognition. Traditional CNNs suffer from expensive computation and high memory consumption, impeding their deployment in real-time recognition systems of SAR sensors, as these systems have low memory resources and low speed of calculation. In this paper, a micro CNN (MCNN) for real-time SAR recognition system is proposed. The proposed MCNN has only two layers, and it is compressed from a deep convolutional neural network (DCNN) with 18 layers by a novel knowledge distillation algorithm called gradual distillation. MCNN is a ternary network, and all its weights are either -1 or 1 or 0. Following a student-teacher paradigm, the DCNN is the teacher network and MCNN is its student network. The gradual distillation makes MCNN a better learning route than traditional knowledge distillation. The experiments on the MSTAR dataset show that the proposed MCNN can obtain a high recognition rate which is almost the same as the DCNN. However, compared with the DCNN, the memory footprint of the proposed MCNN is compressed 177 times, and the calculated amount is 12.8 times less, which means that the proposed MCNN can obtain better performance with the smaller network.

/pdf/a-gradually-distilled-cnn-for-sar-target-recognition-5dnwrn3fn9.pdf

A Gradually Distilled CNN for SAR Target Recognition

Under the framework of a supervised learning-based automatic target recognition (ATR) approach, recognition performance is primarily dependent on the amount of training samples. However, shortage in training samples is a consistent issue for ATR. In this article, we propose a new image to image generation method, called label-directed generative adversarial networks (LDGANs), which will provide labeled samples to be used for recognition model training. We define an entirely new loss function for the LDGAN, which utilizes the Wasserstein distance to replace the original distance measurement of the conventional generative adversarial networks (GANs), thus efficiently avoiding the collapse mode problem. The label information is also added to the loss function of the LDGAN to avoid generating a large number of unlabeled target images. More importantly, the proposed method also makes corresponding changes to the network architecture regarding the new GANs. At the same time, the detailed algorithm about the LDGAN is also introduced in this article to deal with the issue that characteristically GANs are not easy to train. Based on comparisons with other directed generation methods, the experimental results show comparative results of several types of generated images in statistical features, gradient features, classic features of synthetic aperture radar (SAR) targets and the independence from the real image. While demonstrating that the images generated by the LDGAN produced better results using the assumptions of independent and identical distribution, the experiment also explores the performance of the generated image in the ATR. A comparison of these experimental results demonstrates a better way to use the generated image for ATR. The experimental results also prove that the proposed method does have the ability to supplement information for ATR when the training sample information is insufficient.

Zongyong Cui

Papers

Dense Attention Pyramid Networks for Multi-Scale Ship Detection in SAR Images

Image Data Augmentation for SAR Sensor via Generative Adversarial Nets

Open Set Incremental Learning for Automatic Target Recognition

A Gradually Distilled CNN for SAR Target Recognition

LDGAN: A Synthetic Aperture Radar Image Generation Method for Automatic Target Recognition