It is an important topic to effectively perceive the gestures of the elderly and patients, communicate conveniently with the deaf and dumb, and accurately transmit abnormal alarm information through the network, which would have vast applications in the future. Seamlessly communicating among people, including vulnerable groups, is an important sign of social civilization and technological progress. As an important sensor, micro-Doppler (MD) radar has many advantages such as not being restricted by ambient light, long detection distance, no need to consider occlusion, and sensitive detection to slight movements. In recent years, it has been used in various recognition tasks. In this article, we proposed a sign language (SL)/hand gesture recognition method based on a novel discriminative feature, built a measurement system of hand movements using an ultrawideband (UWB) radar, measured ten-type, 15-type SL actions, and ten-type hand gesture actions, and completed the tasks of ten-type SL/hand gesture recognition based on the extracted new features. Compared with the five related methods, experimental results show that the proposed method improves the recognition accuracy by 6.4% on the ten-type SL dataset, which proves the effectiveness of the new cumulative distribution density (CDD) features. For comparison, the proposed method is directly applied to the ten-type hand gesture dataset without any change, and recognition accuracy is improved by 8.6%, which verifies the universality of the proposed new CDD features. Meanwhile, we also give the experimental results when the radial distance measured by radar changes and the types of SL increase.

Sign Language/Gesture Recognition Based on Cumulative Distribution Density Features Using UWB Radar

Sign language is the language of the deaf and mute. However, this particular population of the world is unfortunately overlooked as the majority of the hearing population does not understand sign language. In this paper, an extensive comparative analysis of various gesture recognition techniques involving convolutional neural networks and machine learning algorithms has been discussed and tested for real-time accuracy. Three models: a pre-trained VGG16 with fine-tuning, VGG16 with transfer learning and a hierarchical neural network were analyzed based on a number of trainable parameters. These models were trained on a self-developed dataset consisting images of Indian Sign Language (ISL) representation of all 26 English alphabets. The performance evaluation was based on the practical application of these models, which was simulated by varying lighting and background environments. Out of the three, the hierarchical model outperformed the other two models to give the best accuracy of 98.52% for one-hand and 97% for two-hand gestures. Thereafter, a conversation interface was built in Django using this model for the real-time gesture to speech conversion and vice versa. This publicly accessible interface can be used by anyone who wishes to learn or converse in ISL.

Benchmarking deep neural network approaches for Indian Sign Language recognition

Sign language recognition is considered the most important and challenging application in gesture recognition, involving the fields of pattern recognition, machine learning and computer vision. This is mainly due to the complex visual–gestural nature of sign languages and the availability of few databases and studies related to automatic recognition. This work presents the development and validation of a Brazilian sign language (Libras) public database. The recording protocol describes (1) the chosen signs, (2) the signaller characteristics, (3) the sensors and software used for video acquisition, (4) the recording scenario and (5) the data structure. Provided that these steps are well defined, a database with more than 1000 videos of 20 Libras signs recorded by twelve different people is created using an RGB-D sensor and an RGB camera. Each sign was recorded five times by each signaller. This corresponds to a database with 1200 samples of the following data: (1) RGB video frames, (2) depth, (3) body points and (4) face information. Some approaches using deep learning-based models were applied to classify these signs based on 3D and 2D convolutional neural networks. The best result shows an average accuracy of 93.3%. This paper presents an important contribution for the research community by providing a publicly available sign language dataset and baseline results for comparison.

Development and validation of a Brazilian sign language database for human gesture recognition

Abstract Sign language recognition is challenged by problems, such as accurate tracking of hand gestures, occlusion of hands, and high computational cost. Recently, it has benefited from advancements in deep learning techniques. However, these larger complex approaches cannot manage long-term sequential data and they are characterized by poor information processing and learning efficiency in capturing useful information. To overcome these challenges, we propose an integrated MediaPipe-optimized gated recurrent unit (MOPGRU) model for Indian sign language recognition. Specifically, we improved the update gate of the standard GRU cell by multiplying it by the reset gate to discard the redundant information from the past in one screening. By obtaining feedback from the resultant of the reset gate, additional attention is shown to the present input. Additionally, we replace the hyperbolic tangent activation in standard GRUs with exponential linear unit activation and SoftMax with Softsign activation in the output layer of the GRU cell. Thus, our proposed MOPGRU model achieved better prediction accuracy, high learning efficiency, information processing capability, and faster convergence than other sequential models. 

https://www.nature.com/articles/s41598-022-15998-7.pdf

An integrated mediapipe-optimized GRU model for Indian sign language recognition

Indian sign language alphabet recognition system using CNN with diffGrad optimizer and stochastic pooling

Recognition of sign language by a system has become important to bridge the communication gap between the abled and the Hearing and Speech Impaired people. This paper introduces an efficient algorithm for translating the input hand gesture in Indian Sign Language (ISL) into meaningful English text and speech. The system captures hand gestures through Microsoft Kinect (preferred as the system performance is unaffected by the surrounding light conditions and object colour). The dataset used consists of depth and RGB images (taken using Kinect Xbox 360) with 140 unique gestures of the ISL taken from 21 subjects, which includes single-handed signs, double-handed signs and fingerspelling (signs for alphabets and numbers), totaling to 4600 images. To recognize the hand posture, the hand region is accurately segmented and hand features are extracted using Speeded Up Robust Features, Histogram of Oriented Gradients and Local Binary Patterns. The system ensembles the three feature classifiers trained using Support Vector Machine to improve the average recognition accuracy up to 71.85%. The system then translates the sequence of hand gestures recognized into the best approximate meaningful English sentences. We achieved 100% accuracy for the signs representing 9, A, F, G, H, N and P.

R Akshaya

Papers

A depth-based Indian Sign Language recognition using Microsoft Kinect