X-ray security screening is widely used to maintain aviation/transport security, and its significance poses a particular interest in automated screening systems. This paper aims to review computerised X-ray security imaging algorithms by taxonomising the field into conventional machine learning and contemporary deep learning applications. The first part briefly discusses the classical machine learning approaches utilised within X-ray security imaging, while the latter part thoroughly investigates the use of modern deep learning algorithms. The proposed taxonomy sub-categorises the use of deep learning approaches into supervised, semi-supervised and unsupervised learning, with a particular focus on object classification, detection, segmentation and anomaly detection tasks. The paper further explores well-established X-ray datasets and provides a performance benchmark. Based on the current and future trends in deep learning, the paper finally presents a discussion and future directions for X-ray security imagery.

Towards Automatic Threat Detection: A Survey of Advances of Deep Learning within X-ray Security Imaging

Towards automatic threat detection: A survey of advances of deep learning within X-ray security imaging

In the field of security, baggage-screening with X-rays is used as nondestructive testing for threat object detection. This is a common protocol when inspecting passenger baggage particularly at airports. Unfortunately, the accuracy of such human inspection is around 80–90%, under optimal operator conditions. For this reason, it is quite necessary to assist human inspectors with the aid of computer vision algorithms. This work proposes a deep learning-based methodology designed to detect threat objects in (single spectrum) X-ray baggage scan images. For this purpose, our proposed framework simulates a large number of X-ray images, using a combination of PGGAN (Karras et al. in International conference on learning representations, 2018. https://openreview.net/forum?id=Hk99zCeAb
 ) and superimposition (Mery and Katsaggelos in 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), 2017.
 https://doi.org/10.1109/CVPRW.2017.37
 ) strategies, that are used to train state-of-the-art detection models such as YOLO (Redmon et al. in You only look once: unified, real-time object detection. CoRR abs/1506.02640, 2015. http://arxiv.org/ abs/1506.02640
 ), SSD (Liu et al. in SSD: single shot multibox detector. CoRR abs/1512.02325, 2015. http://arxiv. org/abs/1512.02325
 ) and RetinaNet (Lin et al. in Focal loss for dense object detection. CoRR abs/1708.02002, 2017. http://arxiv.org/abs/1708.02002
 ). Our method has been tested on real X-ray images in the detection of four categories of threat objects: guns, knives, razor blades and shuriken (ninja stars). In our experiments, YOLOv3 (Redmon and Farhadi in Yolov3: An incremental improvement. CoRR abs/1804.02767, 2018. http://arxiv.org/abs/1804.02767
 ) obtained the best mean average precision (mAP) with 96.3% for guns, 76.2% for knives, 86.9% for razor blades and 93.7% for shuriken, while the average mAP for all threat objects was 80.0%. We believe the effectiveness of our method in the detection of threat objects makes its use in checkpoints possible. Moreover, our methodology is scalable and can be easily extended to detect other categories automatically.

Detection of threat objects in baggage inspection with X-ray images using deep learning

According to an aspect, a method comprises: generating a 2D projection from a 3D representation of an object, wherein the 2D projection comprises an edgemapped projection of the 3D representation, generating, with a generative adversarial neural network (GAN), and based on the edgemapped projection, a simulated image of the object, wherein the simulated image appears as though the object has been scanned by a detection device, combining the simulated image of the object with a background image to form a synthesized image, wherein the background image was captured by a detection device, and outputting the synthesized image.

Generating synthetic image data

Automatic security inspection has received increasing interests in recent years. Due to the fixed top-down perspective of X-ray scanning of often tightly packed luggages, such images typically suffer from penetration-induced occlusions, severe object overlapping and violent changes in appearance. For this particular application, few research efforts have been made. To deal with the overlapping in X-ray images classification, we propose a novel Security X-ray Multi-label Classification Network (SXMNet). Our hypothesis is that different overlapping levels and scale variations are the primary challenges in the multi-label classification problem of prohibited items. To address these challenges, we propose to incorporate 1) spatial attention to locate prohibited items despite shape, color and texture variations; and 2) anisotropic fusion of per-stage predictions to dynamically fuse hierarchical visual information under violent variations. Motivated by these, our SXMNet is boosted by bottom-up attention and neural-guided Meta Fusion. Raw input image is exploited to generate high-quality attention masks in a bottom-up way for pyramid feature refinement. Subsequently, the per-stage predictions according to the refined features are automatically re-weighted and fused via a soft selection guided by neural knowledge. Comprehensive experiments on the Security Inspection X-ray (SIXray) and Occluded Prohibited Items X-ray (OPIXray) datasets demonstrate the superiority of the proposed method.

/pdf/multi-label-x-ray-imagery-classification-via-bottom-up-25thsymzoa.pdf

Multi-label X-Ray Imagery Classification via Bottom-Up Attention and Meta Fusion

Motivated by the detection of prohibited objects in carry-on luggage as a part of avionic security screening, we develop a CNN-based object detection approach for multi-view X-ray image data. Our contributions are two-fold. First, we introduce a novel multi-view pooling layer to perform a 3D aggregation of 2D CNN-features extracted from each view. To that end, our pooling layer exploits the known geometry of the imaging system to ensure geometric consistency of the feature aggregation. Second, we introduce an end-to-end trainable multi-view detection pipeline based on Faster R-CNN, which derives the region proposals and performs the final classification in 3D using these aggregated multi-view features. Our approach shows significant accuracy gains compared to single-view detection while even being more efficient than performing single-view detection in each view.

Multi-view X-Ray R-CNN.

Multi-view X-ray R-CNN

Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based models to become multilingual, and -- vice versa -- multilingual models to become multimodal. Our proposed methods outperform current state-of-the-art multilingual multimodal models (e.g., M3P) in zero-shot cross-lingual settings, but the accuracy remains low across the board; a performance drop of around 38 accuracy points in target languages showcases the difficulty of zero-shot cross-lingual transfer for this task. Our results suggest that simple cross-lingual transfer of multimodal models yields latent multilingual multimodal misalignment, calling for more sophisticated methods for vision and multilingual language modeling. The xGQA dataset is available online at: this https URL.

xGQA: Cross-Lingual Visual Question Answering

Reasoning over multiple modalities, e.g. in Visual Question Answering (VQA), requires an alignment of semantic concepts across domains. Despite the widespread success of end-to-end learning, today's multimodal pipelines by and large leverage pre-extracted, fixed features from object detectors, typically Faster R-CNN, as representations of the visual world. The obvious downside is that the visual representation is not specifically tuned to the multimodal task at hand. At the same time, while transformer-based object detectors have gained popularity, they have not been employed in today's multimodal pipelines. We address both shortcomings with TxT, a transformer-based crossmodal pipeline that enables fine-tuning both language and visual components on the downstream task in a fully end-to-end manner. We overcome existing limitations of transformer-based detectors for multimodal reasoning regarding the integration of global context and their scalability. Our transformer-based multimodal model achieves considerable gains from end-to-end learning for multimodal question answering.

Jan-Martin O. Steitz

Papers

Multi-view X-Ray R-CNN.

Multi-view X-ray R-CNN

xGQA: Cross-Lingual Visual Question Answering

TxT: Crossmodal End-to-End Learning with Transformers