Hand gestures have been the key component of communication since the beginning of an era. The hand gestures are the foundation of sign language, which is a visual form of communication. In this paper, a deep learning based convolutional neural network (CNN) model is specifically designed for the recognition of gesture-based sign language. This model has a compact representation that achieves better classification accuracy with a fewer number of model parameters over the other existing architectures of CNN. In order to evaluate the efficacy of this model, VGG-11 and VGG-16 have also been trained and tested in this work. To evaluate the performance, 2 datasets have been considered. First, in this work, a large collection of Indian sign language (ISL) gestures consisting of 2150 images is collected using RGB camera, and second, a publicly available American sign language (ASL) dataset is used. The highest accuracy of 99.96% and 100% is obtained by the proposed model for ISL and ASL datasets respectively. The performance of the proposed system, VGG-11, and VGG-16 are experimentally evaluated and compared with the existing state-of-art approaches. In addition to accuracy, other efficiency indices have been also used to ascertain the robustness of the proposed work. The findings indicate that the proposed model outperforms the existing techniques as it has the potential to classify maximum gestures with a minimal rate of error. The model is also tested with the augmented data and is found as invariant to rotation and scaling transformation.

Vision-based hand gesture recognition using deep learning for the interpretation of sign language

People with speech disabilities communicate in sign language and therefore have trouble in mingling with the able-bodied. There is a need for an interpretation system which could act as a bridge between them and those who do not know their sign language. A functional unobtrusive Indian sign language recognition system was implemented and tested on real world data. A vocabulary of 140 symbols was collected using 18 subjects, totalling 5041 images. The vocabulary consisted mostly of two-handed signs which were drawn from a wide repertoire of words of technical and daily-use origins. The system was implemented using Microsoft Kinect which enables surrounding light conditions and object colour to have negligible effect on the efficiency of the system. The system proposes a method for a novel, low-cost and easy-to-use application, for Indian Sign Language recognition, using the Microsoft Kinect camera. In the fingerspelling category of our dataset, we achieved above 90% recognition rates for 13 signs and 100% recognition for 3 signs with overall 16 distinct alphabets (A, B, D, E, F, G, H, K, P, R, T, U, W, X, Y, Z) recognised with an average accuracy rate of 90.68%.

Nearest neighbour classification of Indian sign language gestures using kinect camera

Sign Language detection by technology is an overlooked concept despite there being a large social group which could benefit by it. There are not many technologies which help in connecting this social group to the rest of the world. Understanding sign language is one of the primary enablers in helping users of sign language communicate with the rest of the society. Image classification and machine learning can be used to help computers recognize sign language, which could then be interpreted by other people. Convolutional neural networks have been employed in this paper to recognize sign language gestures. The image dataset used consists of static sign language gestures captured on an RGB camera. Preprocessing was performed on the images, which then served as the cleaned input. The paper presents results obtained by retraining and testing this sign language gestures dataset on a convolutional neural network model using Inception v3. The model consists of multiple convolution filter inputs that are processed on the same input. The validation accuracy obtained was above 90% This paper also reviews the various attempts that have been made at sign language detection using machine learning and depth data of images. It takes stock of the various challenges posed in tackling such a problem, and outlines future scope as well.

Sign Language Recognition Using Deep Learning on Custom Processed Static Gesture Images

Recognition of sign language by a system has become important to bridge the communication gap between the abled and the Hearing and Speech Impaired people. This paper introduces an efficient algorithm for translating the input hand gesture in Indian Sign Language (ISL) into meaningful English text and speech. The system captures hand gestures through Microsoft Kinect (preferred as the system performance is unaffected by the surrounding light conditions and object colour). The dataset used consists of depth and RGB images (taken using Kinect Xbox 360) with 140 unique gestures of the ISL taken from 21 subjects, which includes single-handed signs, double-handed signs and fingerspelling (signs for alphabets and numbers), totaling to 4600 images. To recognize the hand posture, the hand region is accurately segmented and hand features are extracted using Speeded Up Robust Features, Histogram of Oriented Gradients and Local Binary Patterns. The system ensembles the three feature classifiers trained using Support Vector Machine to improve the average recognition accuracy up to 71.85%. The system then translates the sequence of hand gestures recognized into the best approximate meaningful English sentences. We achieved 100% accuracy for the signs representing 9, A, F, G, H, N and P.

A depth-based Indian Sign Language recognition using Microsoft Kinect

A machine cannot easily understand and interpret three-dimensional (3D) data. In this study, we propose the use of graph matching (GM) to enable 3D motion capture for Indian sign language recognition. The sign classification and recognition problem for interpreting 3D motion signs is considered an adaptive GM (AGM) problem. However, the current models for solving an AGM problem have two major drawbacks. First, spatial matching can be performed on a fixed set of frames with a fixed number of nodes. Second, temporal matching divides the entire 3D dataset into a fixed number of pyramids. The proposed approach solves these problems by employing interframe GM for performing spatial matching and employing multiple intraframe GM for performing temporal matching. To test the proposed model, a 3D sign language dataset is created that involves 200 continuous sentences in the sign language through a motion capture setup with eight cameras.The method is also validated on 3D motion capture benchmark action dataset HDM05 and CMU. We demonstrated that our approach increases the accuracy of recognizing signs in continuous sentences.

Indian sign language recognition using graph matching on 3D motion captured signs

Online commerce platforms connect suppliers to customers using internet. In online travel industry, the suppliers provide a programmatic interface for the platform to query and book travel services. Typically, multiple competitive platforms connect to the same set of suppliers and any change in inventory level leads to changes in availability and pricing. One of the key problems that online platforms deal with is to ensure the accuracy of inventory availability and pricing, while ensuring low response time to the customers. In this work, we present the design and implementation of DynaCorrect, a dynamically correcting cache for non-cooperative systems. DynaCorrect caches inventory data from remote suppliers to ensure low response time. Since the inventory price and availability may change in the background, DynaCorrect employs random sampling and a feedback loop to correct the cache. Further, in order to ensure high cache hit rate, it refreshes inventory based on popularity, spatial and temporal locality. Finally, it employs deduplication to ensure high resource efficiency for the caching system. We implemented DynaCorrect in production over a period of time and observed significant benefits to achieve the goals of high accuracy, low response time and reduced number of queries to the suppliers.

Zafar Ahmed Ansari

Papers

Nearest neighbour classification of Indian sign language gestures using kinect camera

DynaCorrect: Dynamically Correcting Cache for Non-cooperative Systems

Gesture Recognition for Indian Sign Language