This work is devoted to an identity document recognition system design for use in mobile phones and tablets using the computational capabilities of the device itself Key differences are discussed in relation to conservative cloud recognition systems which commonly use single images as an input by design A mobile recognition system chart is presented which is constructed with computational limitations in mind and which is implemented in a commercial solution An original approach designed to improve recognition precision and reliability using post-OCR results integration in video stream, as opposed to approaches which rely on frame image integration using "super-resolution" algorithms An interactive feedback between the system and its operator is discussed, such as automatic video stream recognition stopping decision Experimental results are presented for an implemented commercial system "Smart IDReader" designed for identity documents recognition

Smart IDReader: Document Recognition in Video Stream

In this paper, we introduce an “on the device” text line recognition framework that is designed for mobile or embedded systems. We consider per-character segmentation as a language-independent problem and individual character recognition as a language-dependent one. Thus, the proposed solution is based on two separate artificial neural networks (ANN) and dynamic programming instead of employing image processing methods for the segmentation step or end-to-end ANN. To satisfy the tight constraints on memory size imposed by embedded systems and to avoid overfitting, we employ ANNs with a small number of trainable parameters. The primary purpose of our framework is the recognition of low-quality images of identity documents with complex backgrounds and a variety of languages and fonts. We demonstrate that our solution shows high recognition accuracy on natural datasets even being trained on purely synthetic data. We use MIDV-500 and Census 1961 Project datasets for text line recognition. The proposed method considerably surpasses the algorithmic method implemented in Tesseract 3.05, the LSTM method (Tesseract 4.00), and unpublished method used in the ABBYY FineReader 15 system. Also, our framework is faster than other compared solutions. We show the language-independence of our segmenter with the experiment with Cyrillic, Armenian, and Chinese text lines.

/pdf/two-step-cnn-framework-for-text-line-recognition-in-camera-4xxgl7t7f1.pdf

Two-Step CNN Framework for Text Line Recognition in Camera-Captured Images

A lot of research has been devoted to identity documents analysis and recognition on mobile devices. However, no publicly available datasets designed for this particular problem currently exist. There are a few datasets which are useful for associated subtasks but in order to facilitate a more comprehensive scientific and technical approach to identity document recognition more specialized datasets are required. In this paper we present a Mobile Identity Document Video dataset (MIDV-500) consisting of 500 video clips for 50 different identity document types with ground truth which allows to perform research in a wide scope of document analysis problems. The paper presents characteristics of the dataset and evaluation results for existing methods of face detection, text line recognition, and document fields data extraction. Since an important feature of identity documents is their sensitiveness as they contain personal data, all source document images used in MIDV-500 are either in public domain or distributed under public copyright licenses. 
The main goal of this paper is to present a dataset. However, in addition and as a baseline, we present evaluation results for existing methods for face detection, text line recognition, and document data extraction, using the presented dataset. 
(The dataset is available for download at this ftp URL.)

/pdf/midv-500-a-dataset-for-identity-documents-analysis-and-7kyvme03d8.pdf

MIDV-500: A Dataset for Identity Documents Analysis and Recognition on Mobile Devices in Video Stream

In this paper we discuss the problem of simultaneous document type recognition and projective distortion parameters estimation for the images of ID documents. There are two considered cases. In the first case a video stream captured using mobile devices is processed on the device. The second case considers photos or scanned images which are processed on a server. For each case the requirements are defined for the input data and processing speed. The universal approach is proposed, which allows solving the problem in both cases. The approach is based on representing the image as a constellation of feature points and descriptors, but in order to perform more accurate distortion parameters estimation straight lines and quadrangles are extracted from the input image and used as additional features. Techniques are described which allow to combine matched feature points, lines, and quadrangles to geometric verification using RANSAC. Best alternative selection criteria are proposed along with methods of solution accuracy estimation. The differences between methods of preliminary analysis of the input image and geometric primitives location are discussed in relation to the considered problems. For quality estimation an open dataset MIDV-500 is used, together with its extension for server-side problem version, created in scope of this work. Results show that using lines and quadrangles increase the location accuracy, and the proposed algorithm surpasses previously published works in classification precision and computational performance.

Fast Method of ID Documents Location and Type Identification for Mobile and Server Application

In this paper we study the real-time augmentation - method of increasing variability of training dataset during the learning process. We consider the most common label-preserving deformations, which can be useful in many practical tasks. Due to limitations of existing augmentation tools like increase in learning time or dependence on a specific platform, we developed own real-time augmentation system. Experiments on MNIST and SVHN datasets demonstrated the effectiveness of suggested approach - the quality of the trained models improves, and learning time remains the same as if augmentation was not used.

Effective real-time augmentation of training dataset for the neural networks learning

A lot of research has been devoted to identity documents analysis and recognition on mobile devices. However, no publicly available datasets designed for this particular problem currently exist. There are a few datasets which are useful for associated subtasks but in order to facilitate a more comprehensive scientific and technical approach to identity document recognition more specialized datasets are required. In this paper we present a Mobile Identity Document Video dataset (MIDV-500) consisting of 500 video clips for 50 different identity document types with ground truth which allows to perform research in a wide scope of document analysis problems. The paper presents characteristics of the dataset and evaluation results for existing methods of face detection, text line recognition, and document fields data extraction. Since an important feature of identity documents is their sensitiveness as they contain personal data, all source document images used in MIDV-500 are either in public domain or distributed under public copyright licenses.\n\nThe main goal of this paper is to present a dataset. However, in addition and as a baseline, we present evaluation results for existing methods for face detection, text line recognition, and document data extraction, using the presented dataset.

/pdf/midv-500-a-dataset-for-identity-document-analysis-and-305gri0rcn.pdf

MIDV-500: a dataset for identity document analysis and recognition on mobile devices in video stream

Recognition of identity documents using mobile devices has become a topic of a wide range of computer vision research. The portfolio of methods and algorithms for solving such tasks as face detection, document detection and rectification, text field recognition, and other, is growing, and the scarcity of datasets has become an important issue. One of the openly accessible datasets for evaluating such methods is MIDV-500, containing video clips of 50 identity document types in various conditions. However, the variability of capturing conditions in MIDV-500 did not address some of the key issues, mainly significant projective distortions and different lighting conditions. In this paper we present a MIDV-2019 dataset, containing video clips shot with modern high-resolution mobile cameras, with strong projective distortions and with low lighting conditions. The description of the added data is presented, and experimental baselines for text field recognition in different conditions.

MIDV-2019: challenges of the modern mobile-based document OCR

/pdf/a-method-to-reduce-errors-of-string-recognition-based-on-2hgzqd7d3r.pdf

Konstantin B. Bulatov

Papers

MIDV-500: a dataset for identity document analysis and recognition on mobile devices in video stream

Smart IDReader: Document Recognition in Video Stream

MIDV-500: A Dataset for Identity Documents Analysis and Recognition on Mobile Devices in Video Stream

MIDV-2019: challenges of the modern mobile-based document OCR

A Method to Reduce Errors of String Recognition Based on Combination of Several Recognition Results with Per-Character Alternatives