scispace - formally typeset
Search or ask a question
Book Chapter•DOI•

Finding recurrent patterns from continuous sign language sentences for automated extraction of signs

Sunita Nayak, Kester Duncan1, Sudeep Sarkar1, Barbara L. Loeding2•
University of South Florida1, Florida Polytechnic University2
01 Jan 2012-Journal of Machine Learning Research (Springer, Cham)-Vol. 13, Iss: 1, pp 2589-2615
TL;DR: In this paper, a probabilistic framework is presented to automatically learn recurring signs from multiple sign language video sequences containing the vocabulary of interest, which is robust to the variations produced by adjacent signs.
Abstract: We present a probabilistic framework to automatically learn models of recurring signs from multiple sign language video sequences containing the vocabulary of interest. We extract the parts of the signs that are present in most occurrences of the sign in context and are robust to the variations produced by adjacent signs. Each sentence video is first transformed into a multidimensional time series representation, capturing the motion and shape aspects of the sign. Skin color blobs are extracted from frames of color video sequences, and a probabilistic relational distribution is formed for each frame using the contour and edge pixels from the skin blobs. Each sentence is represented as a trajectory in a low dimensional space called the space of relational distributions. Given these time series trajectories, we extract signemes from multiple sentences concurrently using iterated conditional modes (ICM). We show results by learning single signs from a collection of sentences with one common pervading sign, multiple signs from a collection of sentences with more than one common sign, and single signs from a mixed collection of sentences. The extracted signemes demonstrate that our approach is robust to some extent to the variations produced within a sign due to different contexts. We also show results whereby these learned sign models are used for spotting signs in test sequences.

Summary (3 min read)

Jump to: [1. Introduction] – [2. Relational Distributions] – [3. Problem Formulation] – [3.1 Distance Measure] – [3.2 Parameter Estimation] – [3.3 Sampling Starting Points For ICM] – [4. Experiments And Results] – [4.1 Data Set] – [4.2 Common Pattern Extraction Results] – [4.2.1 EXTRACTING THE MOST COMMON PATTERN] – [4.2.2 EXTRACTING MULTIPLE COMMON SIGNS] – [4.2.3 EXTRACTING THE MOST COMMON PATTERNS FROM MIXED SENTENCES] and [4.3 Sign Localization]

1. Introduction

  • Most of the existing work in sign language assumes that the training signs are already available and often signs used in the training set are the isolated signs with the boundaries chopped off, or manually selected frames from continuous sentences.
  • The process is iterated till the parameter values converge to a stable solution.
  • The authors also extract single signs from a mixed collection of sentences where there are more than one common sign in context.

2. Relational Distributions

  • The authors use relational distributions to capture the global and relative configuration of the hands and the face in an image.
  • The authors start from some level of segmentation of the object from the scene.
  • It captures the global configuration of the low-level primitives.
  • Figure 3(c) illustrates how motion is captured using relational distributions.
  • Each bin then counts the pairs of edge pixels between which the horizontal and vertical distances each lie in some fixed range that depends on the location of the bin in the histogram.

3. Problem Formulation

  • Sign language sentences are series of signs.
  • Figure 4 illustrates the traces of the first vs. second dimension in the feature space, of three sentences S1, S2 and S3 with only one common sign, R, among them.
  • Table 3 defines the notations that will be used in this paper.
  • Also note that p(θ) is hard to compute or even sample from because it is computationally expensive to compute the denominator in Equation 2, as it involves the summation over all possible parameter combinations.
  • In other words, the authors construct a probability density function of the possible starting points and widths in each sentence, given the estimated starting points and widths of the common pattern in all other sentences, that is, f (θi|θ(i)).

3.1 Distance Measure

  • The distance function d in the above equations needs to be chosen carefully such that it is not biased towards the shorter subsequences.
  • Here, the authors briefly describe how they compute the distance between two substrings using dynamic time warping.
  • Let l1 and l2 represent the length of the two substrings and e(i, j) represent the Euclidean distance between the ith data point from the first substring and the jth data point from the second substring.

3.2 Parameter Estimation

  • Gibbs sampling (Casella and George, 1992) is a Markov Chain Monte Carlo approach (Gilks et al., 1998) that allows us to sample the conditional probability density f (θi|θ(i)) for all the sequences sequentially and then iterate the whole process until convergence.
  • ICM has much faster convergence, but it is also known to be heavily dependent on the initialization.
  • The values for ai and wi are updated with those that maximize the conditional density f (θi|θ(i)).
  • The vertical axis in the probabilities represents the starting locations and the horizontal axis represents the possible widths.
  • Note that the probabilities are spread out in the first iteration for each sentence and it slowly converges to a fixed starting location for each of them.

3.3 Sampling Starting Points For ICM

  • In order to address the local convergence nature of ICM, the authors adopt a uniform random samplingbased approach.
  • The value for a0i is obtained by sampling a starting point based on uniform random distribution from the set of all possible starting points in the ith sequence, that is, from the set {1 · · ·(Li−w 0 i +1)}.
  • ICM is run using each initial parameter vector generated and the most common solution is considered as the final solution.
  • The authors run it the number of times equal to the average number of frames in each sentence from the given set of sentences for extracting the sign.
  • Assign most frequently occurring value as the final value for each parameter, also known as comment.

4. Experiments And Results

  • The authors present visual and quantitative results of their approach for extracting signemes from video sequences representing sentences from American Sign Language.
  • The authors first describe the data set used then present the results of the automatic common pattern extraction.

4.1 Data Set

  • The authors data set consists of 155 American Sign Language (ASL) video sequences organized into 12 groups based on the vocabulary (word that pervades the sentences of the group).
  • The breakdown of these ‘pure’ groups and the number of sentences in each are as follows.
  • The initial parameter vector for each ICM run was chosen independently using uniform random sampling.
  • This data set was used to extract 12 common subsequences when the authors searched for the first most common sign, and 24 common subsequences when they searched for the second most common sign.
  • All of the signs were performed by the same signer with plain clothing and background.

4.2 Common Pattern Extraction Results

  • The authors present the results of their method for extracting common patterns from sign language sentences.
  • The authors first present results for extracting the single most common sign and multiple common signs from the ‘pure’ sentence groups, followed by results for the most common patterns from the ‘mixed’ groups.

4.2.1 EXTRACTING THE MOST COMMON PATTERN

  • The authors perform extraction of the most common patterns from the ‘pure’ sentence groups.
  • The authors possess a priori knowledge of the most common word due to the organization of the sentence groups.
  • As can be seen, the extracted patterns and the corresponding ground truth patterns are quite similar, except for a few frames at the beginning and end of the some of the patterns.
  • Figure 10(b) shows the corresponding scatter plot for the end position of the patterns in the sentences.
  • As can be seen most of the points in the scatter plots lie along the diagonal.

4.2.2 EXTRACTING MULTIPLE COMMON SIGNS

  • In this section the authors present some visual results for the extraction of the two most common signs from the ‘pure’ groups of sentences.
  • The authors focused on extracting only two signs because the shortest ASL sentence contained two signs.
  • Figure 13 shows the results for the two most common signs extracted from the sentence ‘BAGGAGE THERE NOT MINE THERE’.
  • The extracted subsequences correspond to the ASL words ‘BAGGAGE’ and ‘MINE’.
  • The word ‘BAGGAGE’ appears in all the 14 sentences of the group, whereas the word ‘MINE’ (or ‘MY’) shows up in 11 sentences coinciding with what was expected.

4.2.3 EXTRACTING THE MOST COMMON PATTERNS FROM MIXED SENTENCES

  • The authors perform extraction of the most common patterns from the collection of ‘mixed’ sentences as outlined in Section 4.1.
  • Figure 15(a) shows the scatter plot of the ground truth start positions vs. the estimated start positions of the pattern extracted from each of the sentences.
  • The frame width range for the sign ‘HAVE’ is between 4 and 6 frames with 4 being the minimum width and 6 being the maximum width.
  • Combining these width ranges could be done using an average of the two or by selecting the minimum and maximum values between the two.

4.3 Sign Localization

  • The same process that is used for training sign models is used for sign localization.
  • The set of points representing the signeme were matched with the segments of the SoRD points from the test sentences to find the segment with the minimum matching score, which would represent the sign in the test sentence.
  • The plot of the Start Offset vs. the End Offset is shown in Figure 16.
  • The points for different signs are scattered in the four quadrants depending on the nature of the overlap between the ground truth sign and the retrieved signeme.
  • The closer it is to the origin, the better the quality.

Did you find this useful? Give us your feedback

Figures (17)

Content maybe subject to copyright    Report

Journal of Machine Learning Research 13 (2012) 2589-2615 Submitted 11/11; Revised 5/12; Published 9/12
Finding Recurrent Patterns from Continuous Sign Language
Sentences for Automated Extraction of Signs
Sunita Nayak SNAYAK@TAAZ.COM
Taaz Inc.
4250 Executive Square, Suite 420
La Jolla, CA 92037 USA
Kester Duncan KKDUNCAN@CSE.USF.EDU
Sudeep Sarkar SARKAR@CSE.USF.EDU
Department of Computer Science & Engineering
University of South Florida
Tampa, FL 33620, USA
Barbara Loeding BARBARA@USF.EDU
Department of Special Education
University of South Florida
Lakeland, FL 33803, USA
Editor: Isabelle Guyon
Abstract
We present a probabilistic framework to automatically learn models of recurring signs from mul-
tiple sign language video sequences containing the vocabulary of interest. We extract the parts of
the signs that are present in most occurrences of the sign in context and are robust to the variations
produced by adjacent signs. Each sentence video is first transformed into a multidimensional time
series representation, capturing the motion and shape aspects of the si gn. Skin color blobs are ex-
tracted from frames of color video sequences, and a probabilistic relational distribution is formed
for each frame using the contour and edge pixels from the skin blobs. Each sentence is represented
as a trajectory in a low dimensional space called the space of relational distributions. Given these
time series trajectories, we extract signemes from multiple sentences concurrently using iterated
conditional modes (ICM). We show results by learning single signs from a collection of sentences
with one common pervading sign, multiple signs from a collection of sentences with more than
one common sign, and single signs from a mixed collection of sentences. The extracted signemes
demonstrate that our approach is robust to some extent to the variations produced within a sign due
to different contexts. We also show results whereby these learned sign models are used for spotting
signs in test sequences.
Keywords: pattern extraction, sign language r ecognition, signeme extraction, sign modeling,
iterated conditional modes
1. Introduction
Sign language research in the computer vision community has primarily focused on improving
recognition rates of signs either by improving the motion representation and similarity measures
(Yang et al., 2002; Al-Jarrah and Halawani, 2001; Athitsos et al., 2004; Cui and Weng, 2000; Wang
et al., 2007; Bauer and Hienz, 2000) or by adding linguistic clues during the recognition process
c
2012 Sunita Nayak, Kester Duncan, Sudeep Sarkar and Barbara Loeding.
NAYAK, DUNCAN, SARKAR AND LOEDING
(Bowden et al., 2004; Derpanis et al., 2004). Ong and Ranganath (2005) presented a review of
the automated sign language research and also highlighted one important issue in continuous sign
language recognition. While signing a sentence, there exists transitions of the hands between two
consecutive signs that do not belong to either sign. This is called movement epenthesis (Liddell and
Johnson, 1989). This needs to be dealt with first before dealing with any other phonological issues
in sign language (Ong and Ranganath, 2005). Most of the existing work in sign language assumes
that the training signs are already available and often signs used in the training set are the isolated
signs with the boundaries chopped off, or manually selected frames from continuous sentences.
The ability to recognize isolated signs does not guarantee the recognition of signs in continuous
sentences. Unlike isolated signs, a sign in a continuous sentence is strongly affected by its context
in the sentence. Figure 1 shows two sentences ‘I BUY TI CKET WHERE?’ and ‘YOU CAN BUY
THIS FOR HER’ with a common sign ‘BUY’ between them. The frames representing the sign
‘BUY’ and the neighboring signs are marked. The unmarked frames between the signs indicate
the frames corresponding to movement epenthesis. It can be observed that the same sign ‘BUY’ is
preceded and succeeded by movement epenthesis that depends on the end and start of the preceding
and succeeding sign respectively. The movement epenthesis also affects how the sign is signed.
This effect makes the automated extraction, modeling and recognition of signs from continuous
sentences more difficult when compared to just plain gestures, isolated signs, or finger spelling.
In this paper, we address the problem of automatically extracting the par t of a sign that is most
common in all occurrences of the sign, and hence expected to be robust with respect to the variation
of adjacent signs. These common parts can be used for spotting or recognition of signs in continuous
sign language sentences. They can also be used by sign language experts for teaching or studying
variations between instances of signs in continuous sign language sentences, or in automated sign
language tutoring systems. Furthermore, they can be used even in the process of translating sign
language videos directly to spoken words.
In a related work inspired by the success of the use of phonemes in speech recognition, the
authors sought to extract common parts in different instances of a sign and thus arrive at a phoneme-
analogue for signs (Bauer and Kraiss, 2002). But unlike speech, sign language does not have a
completely defined set of phonemes. Hence, we consider extracting commonalities at the sentence
and sub-sentence level.
A different but a closely related problem is the extraction of common subsequences, also called
motifs, from very long multiple gene sequences in biology (Bailey and Elkan, 1995; Lawrence et al.,
1993; Pevzner and Sze, 2000; Rigoutsos and Floratos, 1998). Lawrence et al. (1993) used a Gibbs
sampling approach based on discrete matches or mismatches of subsequences that were strings of
symbols of gene sequences. Bailey and Elkan (1995) used expectation maximization to find com-
mon subsequences in univariate biopolymer sequences. In biology, researchers deal with univariate
discrete sequences, and hence their algorithms are not always directly applicable to other multi-
variate continuous domains in time series like speech or sign language. Some researchers tried to
symbolize a continuous time series into discrete sequences and used existing algorithms from bioin-
formatics. For example, Chiu et al. (2003) symbolized the time series into a sequence of symbols
using local approximations and used random projections to extract common subsequences in noisy
data. Tanaka et al. (2005) extended their work by performing principal component analysis on the
multivariate time series data and projected them onto a single dimension and symbolized the data
into discrete sequences. However, it is not always possible to get all the important information in
2590
FINDING RECURRENT PATTERNS FROM CONTINUOUS SIGN LANGUAGE SENTENCES
(a) Continuous Sentence ‘I BUY TICKET WHERE?’
(b) Continuous Sentence ‘YOU CAN BUY THIS FOR HER’
Figure 1: Movement epenthesis in sign language sentences. Frames corresponding to the common
sign ‘BUY’ are marked in red. Signs adjacent to BUY are marked in magenta. Frames
between marked frames represent movement epenthesis that is, the transition between
signs. Note that the sign itself is also affected by having different signs preceding or
following it.
the first principal component alone. Further extending his work, Duchne et al. (2007) find recurrent
patterns from multivariate discrete data using time series random projections.
Due to the inherent continuous nature of many time series data like gesture and speech, new
methods were developed that do not require approximating the data to a sequence of discrete sym-
bols. Denton (2005) used a continuous random-walk noise model to cluster similar substrings.
Nayak et al. (2005) and Minnen et al. (2007) use continuous multivariate sequences and dynamic
time warping to find distances between the substrings. Oates (2002); Nayak et al. (2005) and Nayak
et al. (2009a) are among the few works in finding recurrent patterns that address non-uniform sam-
pling of time series. The recurrent pattern extraction approach proposed in this paper is based
2591
NAYAK, DUNCAN, SARKAR AND LOEDING
on multivariate continuous time series, uses dynamic time warping to find distances between sub-
strings, and handles length variations of common patterns.
Following the success of Hidden Markov Models (HMMs) in speech recognition, they were
used by sign language researchers (Vogler and Metaxas, 1999; Starner and Pentland, 1997; Bowden
et al., 2004; Bauer and Hienz, 2000; Starner et al., 1998) for representing and recognizing signs.
However, HMMs require a large number of training data and unlike speech, data from native sign-
ers is not as easily available as speech data. Hence, non-HMM-based approaches have been used
(Farhadi et al., 2007; Nayak et al., 2009a; Yang et al., 2010; Buehler et al., 2009; Nayak et al.,
2009b; Oszust and Wysocki, 2010; Han et al., 2009). In this paper, we use a continuous trajectory
representation of signs in a multidimensional space and use dynamic time warping to match sub-
sequences. The relative configuration of the two hands and face in each frame is represented by a
relational distribution (Vega and Sarkar, 2003; Nayak et al., 2005), which in itself is a probability
density function. The motion dynamics of the s igner is captured as changes in the relational distri-
butions. It also allows us to interpolate motion, if required, for data sets with lower frame capture
rates. It should also be noted that, unlike many of the previous works in sign language that perform
tracking of the hands using 3D magnetic trackers or color gloves (Fang et al., 2004; Vogler and
Metaxas, 2001; Wang et al., 2002; Ma et al., 2000; Cooper and Bowden, 2009), our representation
does not require tracking and relies on skin segmentation.
We present a Bayesian framework to extract the common subsequences or signemes from all
the given sentences simultaneously. Figure 2 depicts the overview of our approach. With this
framework, we can extract the first most common sign, the second most common sign, the third
most common sign and so on. We represent each sentence as a trajectory in a multi-dimensional
space that implicitly captures the shape and motion in the video. Skin color blobs are extracted
from frames of color video, and a relational distribution is formed for each frame using the edge
pixels in the skin blobs. Each sentence is then represented as a trajectory in a low dimensional space
called the space of relational distributions, which is arrived at by performing principal component
analysis (PCA) on the relational distributions. There are other alternatives to PCA that are possible
and discussed in Nayak et al. (2009b). The other choices do not change the nature of the signeme
finding approach, they only affect the quality of the features. The starting locations (a
1
,...a
n
) and
widths (w
1
,...w
n
) of the candidate signemes in all the n sentences are together represented by a
parameter vector. The starting locations are initialized with random starting locations, based on
uniform random sampling from each sentence, and the initial width values are randomly selected
from a given range of values. The parameter vector is updated sequentially by sampling the starting
point and width of the possible signeme in each sentence from a joint conditional distribution that is
based on the locations and widths of the target possible signeme in all other sentences. The process
is iterated till the parameter values converge to a stable solution. Monte Carlo approaches like
Gibbs sampling (Robert and Casella, 2004; Gilks et al., 1998; Casella and George, 1992), which
is a special case of the Metropolis-Hastings algorithm (Chib and Greenberg, 1995) can be used for
global optimization while updating the parameter vector by performing importance sampling on the
conditional probability distribution. However, this has a high burn-in period.
In this paper, we adopt a greedy approach based on the us e of iterated conditional modes (ICM)
(Besag, 1986). ICM converges much faster than a Gibbs sampler, but is known to be largely de-
pendent on the initialization. We overcome this limitation by performing ICM a number of times
equal to the average length of the n sentences, with different initializations. The most frequently
occurring solution from all the ICM runs is considered as the final solution.
2592
FINDING RECURRENT PATTERNS FROM CONTINUOUS SIGN LANGUAGE SENTENCES
Figure 2: Overview of our approach. Each of the n sentences is represented as a sequence in the
Space of Relational Distributions, and common patterns are extracted using iterated con-
ditional modes (ICM). The parameter set {a
1
,w
1
,...a
n
,w
n
} is initialized using uniform
random sampling and the conditional density corresponding to each sentence is updated
in a sequential manner.
The work in this paper builds on the work of Nayak et al. (2009a) and is different in multiple
respects. We propose a system that is generalized to extract more than one common sign from a
collection of sentences (first most common sign, second most common sign and so on), whereas
2593
Citations
More filters
Automatic Analysis of Facial Actions: Learning from Transductive, Supervised and Unsupervised Frameworks

[...]

Wen-Sheng Chu
01 Jan 2017
TL;DR: A transductive framework, Selective Transfer Machine (STM), which personalizes generic classifiers through joint sample reweighting and classifier learning and offers improved generalization to unknown persons is proposed, and a variant of STM is developed for use when partially labeled data are available.
Abstract: Automatic analysis of facial actions (AFA) can reveal a person’s emotion, intention, and physical state, and make possible a wide range of applications. To enable reliable, valid, and efficient AFA, this thesis investigates automatic analysis of facial actions through transductive, supervised and unsupervised learning. Supervised learning for AFA is challenging, in part, because of individual differences among persons in face shape and appearance and variation in video acquisition and context. To improve generalizability across persons, we propose a transductive framework, Selective Transfer Machine (STM), which personalizes generic classifiers through joint sample reweighting and classifier learning. By personalizing classifiers, STM offers improved generalization to unknown persons. As an extension, we develop a variant of STM for use when partially labeled data are available. Additional challenges for supervised learning include learning an optimal representation for classification, variation in base rates of action units (AUs), correlation between AUs and temporal consistency. While these challenges could be partly accommodated with an SVM or STM, a more powerful alternative is afforded by an end-to-end supervised framework (i.e., deep learning). We propose a convolutional network with long short-term memory (LSTM) and multi-label sampling strategies. We compared SVM, STM and deep learning approaches with respect to AU occurrence and intensity in and between BP4D+ [282] and GFT [93] databases, which consist of around 0.6 million annotated frames. Annotated video is not always possible or desirable. We introduce an unsupervised Branch-and-Bound framework to discover correlated facial actions in un-annotated video. We term this approach Common Event Discovery (CED). We evaluate CED in video and motion capture data. CED achieved moderate convergence with supervised approaches and enabled discovery of novel patterns occult to supervised approaches.

3 citations

Book Chapter•DOI•
Unsupervised Discovery of Sign Terms by K-Nearest Neighbours Approach

[...]

Korhan Polat1, Murat Saraclar1•
Boğaziçi University1
23 Aug 2020
TL;DR: In this article, the authors used visual features extracted from RGB videos to find the repeating terms from continuous sign videos without any supervision, using a k-nearest neighbours based discovery algorithm designed for speech can also discover sign terms.
Abstract: In order to utilize the large amount of unlabeled sign language resources, unsupervised learning methods are needed. Motivated by the successful results of unsupervised term discovery (UTD) in spoken languages, here we explore how to apply similar methods for sign terms discovery. Our goal is to find the repeating terms from continuous sign videos without any supervision. Using visual features extracted from RGB videos, we show that a k-nearest neighbours based discovery algorithm designed for speech can also discover sign terms. We also run experiments using a baseline UTD algorithm and comment on their differences.

3 citations

Proceedings Article•DOI•
Providing Meaningful Alignments for Periodic Signs

[...]

Alex Dillhoff1, Himanshu Pahwa1, Christopher Conly1, Vassilis Athitsos1•
University of Texas at Arlington1
21 Jun 2017
TL;DR: This work evaluates two DTW-based approaches, one of which uses a newly defined periodic warping path and the other uses manual annotations to truncate periodic input to contain no more than two periods.
Abstract: In sign languages, a periodic sign is one that contains repeated movements. Dynamic Time Warping (DTW) is often used in sign language recognition to generate a frame alignment between two input signs that provides a measure of their similarity. Alignments provided by DTW may not be meaningful when the input contains periodic signs, especially when the number of periods differs between inputs. Additionally, the number of periods may change between individual signers and signs. Little work has been done to address the problem of recognizing periodic signs in the context of DTW. This work evaluates two DTW-based approaches. The first uses a newly defined periodic warping path. The second uses manual annotations to truncate periodic input to contain no more than two periods. These two methods are compared against a standard implementation of DTW. Recognition accuracy and quality of alignment are analyzed. The results motivate a need for further research in periodic sign language recognition.

1 citations

Spatial and temporal modelling for automatic human behavioral analysis

[...]

Ruiqi Zhao1•
Ohio State University1
01 Jan 2016
TL;DR: This thesis studies two tasks for automatic human behavioral analysis from video: recovering 3D shape from 2D landmarks on a single image and action recognition from video and proposes efficient and accurate methods for these tasks.
Abstract: Human behavioral analysis from video is a topic of major interest in computer vision. It plays key role in a wide range of human-centric applications. A working technique involves processing both static visual cues and temporal dynamics of the entire sequence. In this thesis, we study two tasks for automatic human behavioral analysis from video: recovering 3D shape from 2D landmarks on a single image and action recognition from video. We propose efficient and accurate methods for these tasks. Estimating 3D shape from 2D landmarks not only provides richer information about objects such as human and face but also benefits other higher level tasks. It is a difficult and ill-posed problem. Previous approaches represent shape deformation using a linear combination of learned 3D shape bases. The linear assumption limits their applicability on human pose which is highly deformable and articulated. With the available 3D ground-truth of a number of 2D samples, we propose a deep network to directly estimate the mapping function from 2D landmark points to their depth. The 3D shape is recovered by combing the input and output of the neural network and is up to a scaling factor. The system is robust to noise and missing data. It runs at > 1, 000 fps during testing and outperforms state-of-the-art by up to two-fold. Recognizing action from video is an important step towards automatic human behavioral analysis. It is a very difficult problem because of many challenges including

1 citations

Journal Article•DOI•
A Comprehensive Review of Sign Language Recognition: Different Types, Modalities, and Datasets

[...]

M. Madhiarasan, Partha Pratim Roy
07 Apr 2022-arXiv.org
TL;DR: This review paper facilitates a comprehensive overview of SLR and discusses the needs, challenges, and problems associated with SLR, and finds the research gap and limitations in this domain and suggest future directions.
Abstract: A machine can understand human activities, and the meaning of signs can help overcome the communication barriers between the inaudible and ordinary people. Sign Language Recognition (SLR) is a fascinating research area and a crucial task concerning computer vision and pattern recognition. Recently, SLR usage has increased in many applications, but the environment, background image resolution, modalities, and datasets affect the performance a lot. Many researchers have been striving to carry out generic real-time SLR models. This review paper facilitates a comprehensive overview of SLR and discusses the needs, challenges, and problems associated with SLR. We study related works about manual and non-manual, various modalities, and datasets. Research progress and existing stateof-the-art SLR models over the past decade have been reviewed. Finally, we find the research gap and limitations in this domain and suggest future directions. This review paper will be helpful for readers and researchers to get complete guidance about SLR and the progressive design of the state-of-the-art SLR model.

1 citations

References
More filters
Book•DOI•
Markov Chain Monte Carlo in Practice

[...]

Walter R. Gilks1, Sylvia Richardson, David Spiegelhalter•
Novartis1
01 Aug 1997-Technometrics
TL;DR: The Markov Chain Monte Carlo Implementation Results Summary and Discussion MEDICAL MONITORING Introduction Modelling Medical Monitoring Computing Posterior Distributions Forecasting Model Criticism Illustrative Application Discussion MCMC for NONLINEAR HIERARCHICAL MODELS.
Abstract: INTRODUCING MARKOV CHAIN MONTE CARLO Introduction The Problem Markov Chain Monte Carlo Implementation Discussion HEPATITIS B: A CASE STUDY IN MCMC METHODS Introduction Hepatitis B Immunization Modelling Fitting a Model Using Gibbs Sampling Model Elaboration Conclusion MARKOV CHAIN CONCEPTS RELATED TO SAMPLING ALGORITHMS Markov Chains Rates of Convergence Estimation The Gibbs Sampler and Metropolis-Hastings Algorithm INTRODUCTION TO GENERAL STATE-SPACE MARKOV CHAIN THEORY Introduction Notation and Definitions Irreducibility, Recurrence, and Convergence Harris Recurrence Mixing Rates and Central Limit Theorems Regeneration Discussion FULL CONDITIONAL DISTRIBUTIONS Introduction Deriving Full Conditional Distributions Sampling from Full Conditional Distributions Discussion STRATEGIES FOR IMPROVING MCMC Introduction Reparameterization Random and Adaptive Direction Sampling Modifying the Stationary Distribution Methods Based on Continuous-Time Processes Discussion IMPLEMENTING MCMC Introduction Determining the Number of Iterations Software and Implementation Output Analysis Generic Metropolis Algorithms Discussion INFERENCE AND MONITORING CONVERGENCE Difficulties in Inference from Markov Chain Simulation The Risk of Undiagnosed Slow Convergence Multiple Sequences and Overdispersed Starting Points Monitoring Convergence Using Simulation Output Output Analysis for Inference Output Analysis for Improving Efficiency MODEL DETERMINATION USING SAMPLING-BASED METHODS Introduction Classical Approaches The Bayesian Perspective and the Bayes Factor Alternative Predictive Distributions How to Use Predictive Distributions Computational Issues An Example Discussion HYPOTHESIS TESTING AND MODEL SELECTION Introduction Uses of Bayes Factors Marginal Likelihood Estimation by Importance Sampling Marginal Likelihood Estimation Using Maximum Likelihood Application: How Many Components in a Mixture? Discussion Appendix: S-PLUS Code for the Laplace-Metropolis Estimator MODEL CHECKING AND MODEL IMPROVEMENT Introduction Model Checking Using Posterior Predictive Simulation Model Improvement via Expansion Example: Hierarchical Mixture Modelling of Reaction Times STOCHASTIC SEARCH VARIABLE SELECTION Introduction A Hierarchical Bayesian Model for Variable Selection Searching the Posterior by Gibbs Sampling Extensions Constructing Stock Portfolios With SSVS Discussion BAYESIAN MODEL COMPARISON VIA JUMP DIFFUSIONS Introduction Model Choice Jump-Diffusion Sampling Mixture Deconvolution Object Recognition Variable Selection Change-Point Identification Conclusions ESTIMATION AND OPTIMIZATION OF FUNCTIONS Non-Bayesian Applications of MCMC Monte Carlo Optimization Monte Carlo Likelihood Analysis Normalizing-Constant Families Missing Data Decision Theory Which Sampling Distribution? Importance Sampling Discussion STOCHASTIC EM: METHOD AND APPLICATION Introduction The EM Algorithm The Stochastic EM Algorithm Examples GENERALIZED LINEAR MIXED MODELS Introduction Generalized Linear Models (GLMs) Bayesian Estimation of GLMs Gibbs Sampling for GLMs Generalized Linear Mixed Models (GLMMs) Specification of Random-Effect Distributions Hyperpriors and the Estimation of Hyperparameters Some Examples Discussion HIERARCHICAL LONGITUDINAL MODELLING Introduction Clinical Background Model Detail and MCMC Implementation Results Summary and Discussion MEDICAL MONITORING Introduction Modelling Medical Monitoring Computing Posterior Distributions Forecasting Model Criticism Illustrative Application Discussion MCMC FOR NONLINEAR HIERARCHICAL MODELS Introduction Implementing MCMC Comparison of Strategies A Case Study from Pharmacokinetics-Pharmacodynamics Extensions and Discussion BAYESIAN MAPPING OF DISEASE Introduction Hypotheses and Notation Maximum Likelihood Estimation of Relative Risks Hierarchical Bayesian Model of Relative Risks Empirical Bayes Estimation of Relative Risks Fully Bayesian Estimation of Relative Risks Discussion MCMC IN IMAGE ANALYSIS Introduction The Relevance of MCMC to Image Analysis Image Models at Different Levels Methodological Innovations in MCMC Stimulated by Imaging Discussion MEASUREMENT ERROR Introduction Conditional-Independence Modelling Illustrative examples Discussion GIBBS SAMPLING METHODS IN GENETICS Introduction Standard Methods in Genetics Gibbs Sampling Approaches MCMC Maximum Likelihood Application to a Family Study of Breast Cancer Conclusions MIXTURES OF DISTRIBUTIONS: INFERENCE AND ESTIMATION Introduction The Missing Data Structure Gibbs Sampling Implementation Convergence of the Algorithm Testing for Mixtures Infinite Mixtures and Other Extensions AN ARCHAEOLOGICAL EXAMPLE: RADIOCARBON DATING Introduction Background to Radiocarbon Dating Archaeological Problems and Questions Illustrative Examples Discussion Index

7,399 citations

Book•
Monte Carlo Statistical Methods

[...]

Christian P. Robert1, George Casella1•
University of Florida1
01 Jan 1999
TL;DR: This new edition contains five completely new chapters covering new developments and has sold 4300 copies worldwide of the first edition (1999).
Abstract: We have sold 4300 copies worldwide of the first edition (1999). This new edition contains five completely new chapters covering new developments.

6,884 citations

Journal Article•DOI•
On the statistical analysis of dirty pictures

[...]

Julian Besag1•
Durham University1
01 Jul 1986-Journal of the royal statistical society series b-methodological
TL;DR: In this paper, the authors proposed an iterative method for scene reconstruction based on a non-degenerate Markov Random Field (MRF) model, where the local characteristics of the original scene can be represented by a nondegenerate MRF and the reconstruction can be estimated according to standard criteria.
Abstract: may 7th, 1986, Professor A. F. M. Smith in the Chair] SUMMARY A continuous two-dimensional region is partitioned into a fine rectangular array of sites or "pixels", each pixel having a particular "colour" belonging to a prescribed finite set. The true colouring of the region is unknown but, associated with each pixel, there is a possibly multivariate record which conveys imperfect information about its colour according to a known statistical model. The aim is to reconstruct the true scene, with the additional knowledge that pixels close together tend to have the same or similar colours. In this paper, it is assumed that the local characteristics of the true scene can be represented by a nondegenerate Markov random field. Such information can be combined with the records by Bayes' theorem and the true scene can be estimated according to standard criteria. However, the computational burden is enormous and the reconstruction may reflect undesirable largescale properties of the random field. Thus, a simple, iterative method of reconstruction is proposed, which does not depend on these large-scale characteristics. The method is illustrated by computer simulations in which the original scene is not directly related to the assumed random field. Some complications, including parameter estimation, are discussed. Potential applications are mentioned briefly.

4,490 citations

Journal Article•DOI•
Understanding the Metropolis-Hastings Algorithm

[...]

Siddhartha Chib1, Edward Greenberg1•
Washington University in St. Louis1
01 Nov 1995-The American Statistician
TL;DR: A detailed, introductory exposition of the Metropolis-Hastings algorithm, a powerful Markov chain method to simulate multivariate distributions, and a simple, intuitive derivation of this method is given along with guidance on implementation.
Abstract: We provide a detailed, introductory exposition of the Metropolis-Hastings algorithm, a powerful Markov chain method to simulate multivariate distributions. A simple, intuitive derivation of this method is given along with guidance on implementation. Also discussed are two applications of the algorithm, one for implementing acceptance-rejection sampling when a blanketing function is not available and the other for implementing the algorithm with block-at-a-time scans. In the latter situation, many different algorithms, including the Gibbs sampler, are shown to be special cases of the Metropolis-Hastings algorithm. The methods are illustrated with examples.

3,886 citations

Journal Article•DOI•
Explaining the Gibbs Sampler

[...]

George Casella1, Edward I. George2•
Cornell University1, University of Texas at Austin2
01 Aug 1992-The American Statistician
TL;DR: A simple explanation of how and why the Gibbs sampler works is given and analytically establish its properties in a simple case and insight is provided for more complicated cases.
Abstract: Computer-intensive algorithms, such as the Gibbs sampler, have become increasingly popular statistical tools, both in applied and theoretical work. The properties of such algorithms, however, may sometimes not be obvious. Here we give a simple explanation of how and why the Gibbs sampler works. We analytically establish its properties in a simple case and provide insight for more complicated cases. There are also a number of examples.

2,656 citations

Related Papers (5)
Sign language recognition using sub-units

[...]

01 Jan 2012-Journal of Machine Learning Research
Helen Cooper, Eng-Jon Ong, Nicolas Pugeault, Richard Bowden 
Learning signs from subtitles: A weakly supervised approach to sign language recognition

[...]

20 Jun 2009
Helen Cooper, Richard Bowden
The American Sign Language Lexicon Video Dataset

[...]

23 Jun 2008
Vassilis Athitsos, Carol Neidle, Stan Sclaroff, Joan Nash, Alexandra Stefan, Quan Yuan, Ashwin Thangali 
Automated sign language recognition

[...]

02 May 2017
Sabri A. Mahmoud, Ala addin I. Sidig
Advances in phonetics-based sub-unit modeling for transcription alignment and sign language recognition

[...]

20 Jun 2011
Vassilis Pitsikalis, Stavros Theodorakis, Christian Vogler, Petros Maragos 
Frequently Asked Questions (2)
Q1. What are the contributions in "Finding recurrent patterns from continuous sign language sentences for automated extraction of signs" ?

The authors present a probabilistic framework to automatically learn models of recurring signs from multiple sign language video sequences containing the vocabulary of interest. The authors extract the parts of the signs that are present in most occurrences of the sign in context and are robust to the variations produced by adjacent signs. Given these time series trajectories, the authors extract signemes from multiple sentences concurrently using iterated conditional modes ( ICM ). The authors show results by learning single signs from a collection of sentences with one common pervading sign, multiple signs from a collection of sentences with more than one common sign, and single signs from a mixed collection of sentences. The extracted signemes demonstrate that their approach is robust to some extent to the variations produced within a sign due to different contexts. The authors also show results whereby these learned sign models are used for spotting signs in test sequences. 

Additionally, the authors plan to extend their work to address the challenge of handling the large variations encountered when automatically recognizing signemes across different signers. The authors plan to work on a variation of dynamic time warping that is robust to amplitude differences between various instances of signs.