Enhance to read better: A Multi-Task Adversarial Network for Handwritten Document Image Enhancement

Document images can be affected by many degradation scenarios, which cause recognition and processing difficulties. In this age of digitization, it is important to denoise them for proper usage. To address this challenge, we present a new encoder-decoder architecture based on vision transformers to enhance both machine-printed and handwritten document images, in an end-to-end fashion. The encoder operates directly on the pixel patches with their positional information without the use of any convolutional layers, while the decoder reconstructs a clean image from the encoded patches. Conducted experiments show a superiority of the proposed model compared to the state-of-the-art methods on several DIBCO benchmarks. Code and models will be publicly available at: https://github.com/dali92002/DocEnTR.

DocEnTr: An End-to-End Document Image Enhancement Transformer

Mask usage is one of the most important precautions to limit the spread of COVID-19. Therefore, hygiene rules enforce the correct use of face coverings. Automated mask usage classification might be used to improve compliance monitoring. This study deals with the problem of inappropriate mask use. To address that problem, 2075 face mask usage images were collected. The individual images were labeled as either mask, no masked, or improper mask. Based on these labels, the following three cases were created: Case 1: mask versus no mask versus improper mask, Case 2: mask versus no mask + improper mask, and Case 3: mask versus no mask. This data was used to train and test a hybrid deep feature-based masked face classification model. The presented method comprises of three primary stages: (i) pre-trained ResNet101 and DenseNet201 were used as feature generators; each of these generators extracted 1000 features from an image; (ii) the most discriminative features were selected using an improved RelieF selector; and (iii) the chosen features were used to train and test a support vector machine classifier. That resulting model attained 95.95%, 97.49%, and 100.0% classification accuracy rates on Case 1, Case 2, and Case 3, respectively. Having achieved these high accuracy values indicates that the proposed model is fit for a practical trial to detect appropriate face mask use in real time.

/pdf/hybrid-deep-feature-generation-for-appropriate-face-mask-use-1xeqnxtg.pdf

Hybrid Deep Feature Generation for Appropriate Face Mask Use Detection

Document images can be affected by many degradation scenarios, which cause recognition and processing difficulties. In this age of digitization, it is important to denoise them for proper usage. To address this challenge, we present a new encoder-decoder architecture based on vision transformers to enhance both machine-printed and handwritten document images, in an end-to-end fashion. The encoder operates directly on the pixel patches with their positional information without the use of any convolutional layers, while the decoder reconstructs a clean image from the encoded patches. Conducted experiments show a superiority of the proposed model compared to the state-of-the-art methods on several DIBCO benchmarks. Code and models will be publicly available at: https://github.com/dali92002/DocEnTR. 

/pdf/docentr-an-end-to-end-document-image-enhancement-transformer-2ac5fjt3.pdf

Automatic document content processing is affected by artifacts caused by the shape of the paper, non-uniform and diverse color of lighting conditions. Fully-supervised methods on real data are impossible due to the large amount of data needed. Hence, the current state of the art deep learning models are trained on fully or partially synthetic images. However, document shadow or shading removal results still suffer because: (a) prior methods rely on uniformity of local color statistics, which limit their application on real-scenarios with complex document shapes and textures and; (b) synthetic or hybrid datasets with non-realistic, simulated lighting conditions are used to train the models. In this paper we tackle these problems with our two main contributions. First, a physically constrained learning-based method that directly estimates document reflectance based on intrinsic image formation which generalizes to challenging illumination conditions. Second, a new dataset that clearly improves previous synthetic ones, by adding a large range of realistic shading and diverse multi-illuminant conditions, uniquely customized to deal with documents in-the-wild. The proposed architecture works in a self-supervised manner where only the synthetic texture is used as a weak training signal (obviating the need for very costly ground truth with disentangled versions of shading and reflectance). The proposed approach leads to a significant generalization of document reflectance estimation in real scenes with challenging illumination. We extensively evaluate on the real benchmark datasets available for intrinsic image decomposition and document shadow removal tasks. Our reflectance estimation scheme, when used as a pre-processing step of an OCR pipeline, shows a 26% improvement of character error rate (CER), thus, proving the practical applicability.

Intrinsic Decomposition of Document Images In-the-Wild.

Shadows of document images bring about difficulties for digitization application and uncomfortable perception in vision. This paper proposes an effective method to remove shadows from the single document images, which contains two stages: shadow detection and shadow removal. For the shadow detection, an iterative neighboring information-based approach is designed to estimate pixel-wise local background color image, which can be used to generate a shadow map. For the shadow removal, a global reference background color is obtained from the local background color image. The shadow scale, which is applied to relight shadowed regions, can be estimated by local and global background color. Moreover, a tone fine-tuning process is designed to make the output colors be normal values. Experiments on document images indicate that the proposed method can produce high-quality unshad-owed document images with a comparable efficiency.

An Effective Background Estimation Method for Shadows Removal of Document Images

Coronavirus disease 2019 (COVID-19) continues to pose a great challenge to the world since its outbreak. To fight against the disease, a series of artificial intelligence (AI) techniques are developed and applied to real-world scenarios such as safety monitoring, disease diagnosis, infection risk assessment, and lesion segmentation of COVID-19 CT scans. The coronavirus epidemics have forced people wear masks to counteract the transmission of virus, which also brings difficulties to monitor large groups of people wearing masks. In this article, we primarily focus on the AI techniques of masked facial detection and related datasets. We survey the recent advances, beginning with the descriptions of masked facial detection datasets. A total of 13 available datasets are described and discussed in detail. Then, the methods are roughly categorized into two classes: conventional methods and neural network-based methods. The conventional methods are usually trained by boosting algorithms with hand-crafted features, which accounts for a small proportion. Neural network-based methods are further classified as three parts according to the number of processing stages. Representative algorithms are described in detail, coupled with some typical techniques that are described briefly. Finally, we summarize the recent benchmarking results, give the discussions on the limitations of datasets and methods, and expand future research directions. To our knowledge, this is the first survey about masked facial detection methods and datasets. Hopefully our survey could provide some help to fight against epidemics.

A Survey on Masked Facial Detection Methods and Datasets for Fighting Against COVID-19

With smart portable devices such as smartphones and tablets in usage and popularity, people are more willing to use these devices to scan and save digitized documents. However, when capturing document images, shadows are inevitable and influence clarity and readability. How to remove the shadows of document images is an important and meaningful task. In this paper, we propose a water-filling method using chroma adjustment for shadow removal. Firstly, a global and local jointly water-filling approach is designed to estimate the shading map. Then, we design an adaptive global brightness adjustment strategy to optimize the global luminance of the output image. Since only adjusting brightness can cause color distortion of output images, we propose an adaptive chroma adjustment strategy to ensure color consistency across all areas of output images. A series of experiments show that our method can remove shadows of digitized documents, outperforming some state-of the-art methods. Moreover, the proposed method can keep the brightness and color as consistent as possible with the non-shadow area.

Joint Water-Filling Algorithm with Adaptive Chroma Adjustment for Shadow Removal From Text Document Images

COVID-19 and its variants have been posing a large risk to people around the world since the outbreak of the disease. Many techniques like AI are explored to help combat epidemics. People are required or forced to wear a mask to fight against COVID-19 epidemics worldwide. It brings new challenges to the task of masked facial region recognition. When facial regions are occluded by masks, it will result in some failures of face detection algorithms. In this paper, we propose a method to recognize masked faces. It mainly includes three parts. Firstly, the human pose is estimated to produce a series of key points. It is implemented by OpenPose. Secondly, a key-points location strategy is designed to capture the masked facial regions. It can locate the positions of faces accurately. Thirdly, the broad learning system, which is also an incremental learning algorithm, is employed to recognize the classes of candidate regions. Experiments conducted on some datasets shed light on the effectiveness of the proposed method.

Masked facial region recognition using human pose estimation and broad learning system

Variable-length license plate segmentation and recognition has always been a challenging barrier in the application of intelligent transportation systems. Previous approaches mainly concern fixed-length license plates, lacking adaptability for variable-length license plates. Although objection detection methods can be used to address the issue, they face a series of difficulties: cross class problem, missing detections, and recognition errors between letters and digits. To solve these problems, we propose a machine learning method that regards each character as a region of interest. It covers three parts. Firstly, we explore a transfer learning algorithm based on Faster-RCNN with InceptionV2 structure to generate candidate character regions. Secondly, a strategy of cross-class removal of character is proposed to reject the overlapped results. A mechanism of template matching and position predicting is designed to eliminate missing detections. Moreover, a twofold broad learning system is designed to identify letters and digits separately. Experiments performed on Macau license plates demonstrate that our method achieves an average 99.68% of segmentation accuracy and an average 99.19% of recognition rate, outperforming some conventional and deep learning approaches. The adaptability is expected to transfer the developed algorithm to other countries or regions.

/pdf/character-segmentation-and-recognition-of-variable-length-2ffzjjyz.pdf

C. L. Philip Chen

Papers

An Effective Background Estimation Method for Shadows Removal of Document Images

A Survey on Masked Facial Detection Methods and Datasets for Fighting Against COVID-19

Joint Water-Filling Algorithm with Adaptive Chroma Adjustment for Shadow Removal From Text Document Images

Masked facial region recognition using human pose estimation and broad learning system

Character Segmentation and Recognition of Variable-Length License Plates Using ROI Detection and Broad Learning System