Showing papers on "Dynamic time warping published in 2019"

Iterative Alignment Network for Continuous Sign Language Recognition

Abstract: In this paper, we propose an alignment network with iterative optimization for weakly supervised continuous sign language recognition. Our framework consists of two modules: a 3D convolutional residual network (3D-ResNet) for feature learning and an encoder-decoder network with connectionist temporal classification (CTC) for sequence modelling. The above two modules are optimized in an alternate way. In the encoder-decoder sequence learning network, two decoders are included, i.e., LSTM decoder and CTC decoder. Both decoders are jointly trained by maximum likelihood criterion with a soft Dynamic Time Warping (soft-DTW) alignment constraint. The warping path, which indicates the possible alignment between input video clips and sign words, is used to fine-tune the 3D-ResNet as training labels with classification loss. After fine-tuning, the improved features are extracted for optimization of encoder-decoder sequence learning network in next iteration. The proposed algorithm is evaluated on two large scale continuous sign language recognition benchmarks, i.e., RWTH-PHOENIX-Weather and CSL. Experimental results demonstrate the effectiveness of our proposed method.

D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation

Abstract: We address weakly supervised action alignment and segmentation in videos, where only the order of occurring actions is available during training. We propose Discriminative Differentiable Dynamic Time Warping (D3TW), the first discriminative model using weak ordering supervision. The key technical challenge for discriminative modeling with weak supervision is that the loss function of the ordering supervision is usually formulated using dynamic programming and is thus not differentiable. We address this challenge with a continuous relaxation of the min-operator in dynamic programming and extend the alignment loss to be differentiable. The proposed D3TW innovatively solves sequence alignment with discriminative modeling and end-to-end training, which substantially improves the performance in weakly supervised action alignment and segmentation tasks. We show that our model is able to bypass the degenerated sequence problem usually encountered in previous work and outperform the current state-of-the-art across three evaluation metrics in two challenging datasets.

A time-series classification approach based on change detection for rapid land cover mapping

Abstract: Land-Use/Land-Cover Time-Series Classification (LULC-TSC) is an important and challenging problem in terrestrial remote sensing. Detecting change-points, dividing the entire time series into multiple invariant subsequences, and classifying the subsequences can improve LULC classification efficiency. Therefore, we have proposed a Time-Series Classification approach based on Change Detection (TSCCD) for rapid LULC mapping that uses the Prophet algorithm to detect the ground-cover change-points and perform time-series segmentation in a time dimension and the DTW (Dynamic Time Warping) algorithm to classify the sub-time series. Since we can assume that the ground cover remains unchanged in each subsequence, only one time-training sample selection and one LULC classification are needed, which greatly improves the work efficiency. Prophet can accurately detect large and subtle changes, capture change direction and change rate, and is strongly robust for handling noise and missing data. DTW is mainly used to improve the accuracy of time-series classification and to resolve the time misalignment problems of ground-cover series data caused by irregular observations or missing values. The results of comparative experiments with BFAST, LandTrendR, and CCDC using simulated time-series showed that TSCCD can detect large and subtle changes and capture change direction and change rate, performing substantially better than the other three contrasting algorithms overall in time-series change detection. Finally, the MODIS (Moderate Resolution Imaging Spectroradiometer) time-series images of Wuhan City from 2000 to 2018 were selected for TSCCD, and the results of China’s national land-use surveys in 2000, 2005, 2008, 2010, 2013, and 2015 were used for cross-validation. The results showed that the classification accuracy of each tested subsequence was higher than 90% and that most Kappa coefficients were greater than 0.9. This means that the proposed TSCCD approach can effectively solve real LULC-TSC problems and has high application value. It can be used for large-area, long time-series LULC classification, which is of great guiding significance for studying global environmental changes, forest-cover changes, and conducting land-use surveys.

Deep Learning for Daily Peak Load Forecasting–A Novel Gated Recurrent Neural Network Combining Dynamic Time Warping

Abstract: Daily peak load forecasting is an essential tool for decision making in power system operation and planning. However, the daily peak load is a nonlinear, nonstationary, and volatile time series, which makes it difficult to be forecasted accurately. This paper, for the first time, proposes a bespoke gated recurrent neural network combining dynamic time warping (DTW) for accurate daily peak load forecasting. The shape-based DTW distance is used to match the most similar load curve, which can capture trends in load changes. By analyzing the relationship between the load curve and the cycle of human social activities, the some-hot encoding scheme is first applied on the discrete variables to expand the features to further characterize their impact on load curves. Then, a three-layer gated recurrent neural network is developed to forecast daily peak load. The proposed algorithm is implemented on the Theano deep learning platform and tested on the loaded dataset of the European Network on Intelligent Technologies. The simulation results show that the proposed algorithm achieves satisfactory results compared with other algorithms using the same dataset in this paper.

IMU Sensor-Based Hand Gesture Recognition for Human-Machine Interfaces

Abstract: We propose an efficient hand gesture recognition (HGR) algorithm, which can cope with time-dependent data from an inertial measurement unit (IMU) sensor and support real-time learning for various human-machine interface (HMI) applications. Although the data extracted from IMU sensors are time-dependent, most existing HGR algorithms do not consider this characteristic, which results in the degradation of recognition performance. Because the dynamic time warping (DTW) technique considers the time-dependent characteristic of IMU sensor data, the recognition performance of DTW-based algorithms is better than that of others. However, the DTW technique requires a very complex learning algorithm, which makes it difficult to support real-time learning. To solve this issue, the proposed HGR algorithm is based on a restricted column energy (RCE) neural network, which has a very simple learning scheme in which neurons are activated when necessary. By replacing the metric calculation of the RCE neural network with DTW distance, the proposed algorithm exhibits superior recognition performance for time-dependent sensor data while supporting real-time learning. Our verification results on a field-programmable gate array (FPGA)-based test platform show that the proposed HGR algorithm can achieve a recognition accuracy of 98.6% and supports real-time learning and recognition at an operating frequency of 150 MHz.

Multiobjective Learning in the Model Space for Time Series Classification

Abstract: A well-defined distance is critical for the performance of time series classification. Existing distance measurements can be categorized into two branches. One is to utilize handmade features for calculating distance, e.g., dynamic time warping, which is limited to exploiting the dynamic information of time series. The other methods make use of the dynamic information by approximating the time series with a generative model, e.g., Fisher kernel. However, previous distance measurements for time series seldom exploit the label information, which is helpful for classification by distance metric learning. In order to attain the benefits of the dynamic information of time series and the label information simultaneously, this paper proposes a multiobjective learning algorithm for both time series approximation and classification, termed multiobjective model-metric (MOMM) learning. In MOMM, a recurrent network is exploited as the temporal filter, based on which, a generative model is learned for each time series as a representation of that series. The models span a non-Euclidean space, where the label information is utilized to learn the distance metric. The distance between time series is then calculated as the model distance weighted by the learned metric. The network size is also optimized to learn parsimonious representations. MOMM simultaneously optimizes the data representation, the time series model separation, and the network size. The experiments show that MOMM achieves not only superior overall performance on uni/multivariate time series classification but also promising time series prediction performance.

Object-based time-constrained dynamic time warping classification of crops using Sentinel-2

Abstract: The increasing volume of remote sensing data with improved spatial and temporal resolutions generates unique opportunities for monitoring and mapping of crops. We compared multiple single-band and multi-band object-based time-constrained Dynamic Time Warping (DTW) classifications for crop mapping based on Sentinel-2 time series of vegetation indices. We tested it on two complex and intensively managed agricultural areas in California and Texas. DTW is a time-flexible method for comparing two temporal patterns by considering their temporal distortions in their alignment. For crop mapping, using time constraints in computing DTW is recommended in order to consider the seasonality of crops. We tested different time constraints in DTW (15, 30, 45, and 60 days) and compared the results with those obtained by using Euclidean distance or a DTW without time constraint. Best classification results were for time delays of both 30 and 45 days in California: 79.5% for single-band DTWs and 85.6% for multi-band DTWs. In Texas, 45 days was best for single-band DTW (89.1%), while 30 days yielded best results for multi-band DTW (87.6%). Using temporal information from five vegetation indices instead of one increased the overall accuracy in California with 6.1%. We discuss the implications of DTW dissimilarity values in understanding the classification errors. Considering the possible sources of errors and their propagation throughout our analysis, we had combined errors of 22.2% and 16.8% for California and 24.6% and 25.4% for Texas study areas. The proposed workflow is the first implementation of DTW in an object-based image analysis (OBIA) environment and represents a promising step towards generating fast, accurate, and ready-to-use agricultural data products.

Recurrent Adaptation Networks for Online Signature Verification

Abstract: Online signature verification remains a challenging task owing to large intra-individual variability. To tackle this problem, in this paper, we propose to use recurrent neural networks (RNN) for representation learning in the dynamic time warping framework. Metric-based loss functions are designed explicitly to minimize intra-individual variability and enhance inter-individual variability and to guide the RNN in learning discriminative representations for online signatures. An RNN variant-gated auto regressive units-is proposed and shows a better generalization performance in our framework. Furthermore, we interpret the online signature verification problem as a meta-learning problem: one client is considered as one task, therefore, different clients compose the task space. Based on this formulation, we design an end-to-end trainable meta-layer that learns to adapt to different clients, allowing fast adaptation to new clients in the test stage. In addition, a new descriptor-the length-normalized path signature-is proposed to describe online signatures. Our proposed system achieves a state-of-the-art performance on three benchmark datasets, namely, MCYT-100, Mobisig, and e-BioSign.

WristCam: A Wearable Sensor for Hand Trajectory Gesture Recognition and Intelligent Human–Robot Interaction

Abstract: As a promising component for body sensor networks, the wearable sensors for hand gesture recognition have increasingly received great attention in recent years. By interpreting human intentions through hand gestures, the natural human–robot interaction can be realized in the smart home where the youth and the elderly can perform hand gestures to control the household robot or the robotic wheelchair. Here, a wearable wrist-worn camera sensor was shown to recognize hand trajectory gestures. The moving velocity of the user’s hand was deduced from the matched speeded up robust features keypoints of the moving background of the video sequence. Furthermore, the segmentation of continuous gestures was achieved by detecting the predefined gesture starting signal from the hand region of the image, which was segmented by the lazy snapping algorithm. In this paper, 10 types of gestures and 1350 gesture samples collected from 15 subjects at three different scenes were classified by the dynamic time warping algorithm and the results achieved an average recognition accuracy up to 97.6%. Moreover, the practicability of the proposed system was further demonstrated by controlling a cooperative robot to draw letters on paper.

dtwSat: Time-Weighted Dynamic Time Warping for Satellite Image Time Series Analysis in R

Abstract: The opening of large archives of satellite data such as LANDSAT, MODIS and the SENTINELs has given researchers unprecedented access to data, allowing them to better quantify and understand local and global land change. The need to analyze such large data sets has led to the development of automated and semi-automated methods for satellite image time series analysis. However, few of the proposed methods for remote sensing time series analysis are available as open source software. In this paper we present the R package dtwSat. This package provides an implementation of the time-weighted dynamic time warping method for land cover mapping using sequence of multi-band satellite images. Methods based on dynamic time warping are flexible to handle irregular sampling and out-of-phase time series, and they have achieved significant results in time series analysis. Package dtwSat is available from the Comprehensive R Archive Network (CRAN) and contributes to making methods for satellite time series analysis available to a larger audience. The package supports the full cycle of land cover classification using image time series, ranging from selecting temporal patterns to visualizing and assessing the results.

Comparing Similarity Perception in Time Series Visualizations

Abstract: A common challenge faced by many domain experts working with time series data is how to identify and compare similar patterns. This operation is fundamental in high-level tasks, such as detecting recurring phenomena or creating clusters of similar temporal sequences. While automatic measures exist to compute time series similarity, human intervention is often required to visually inspect these automatically generated results. The visualization literature has examined similarity perception and its relation to automatic similarity measures for line charts, but has not yet considered if alternative visual representations, such as horizon graphs and colorfields, alter this perception. Motivated by how neuroscientists evaluate epileptiform patterns, we conducted two experiments that study how these three visualization techniques affect similarity perception in EEG signals. We seek to understand if the time series results returned from automatic similarity measures are perceived in a similar manner, irrespective of the visualization technique; and if what people perceive as similar with each visualization aligns with different automatic measures and their similarity constraints. Our findings indicate that horizon graphs align with similarity measures that allow local variations in temporal position or speed (i.e., dynamic time warping) more than the two other techniques. On the other hand, horizon graphs do not align with measures that are insensitive to amplitude and y-offset scaling (i.e., measures based on z-normalization), but the inverse seems to be the case for line charts and colorfields. Overall, our work indicates that the choice of visualization affects what temporal patterns we consider as similar, i.e., the notion of similarity in time series is not visualization independent.

Usage of autoencoders and Siamese networks for online handwritten signature verification

Abstract: In this paper, we propose a novel writer-independent global feature extraction framework for the task of automatic signature verification which aims to make robust systems for automatically distinguishing negative and positive samples. Our method consists of an autoencoder for modeling the sample space into a fixed-length latent space and a siamese network for classifying the fixed-length samples obtained from the autoencoder based on the reference samples of a subject as being genuine or forged. During our experiments, usage of attention mechanism and applying downsampling significantly improved the accuracy of the proposed framework. We evaluated our proposed framework using SigWiComp2013 Japanese and GPDSsyntheticOnLineOffLineSignature datasets. On the SigWiComp2013 Japanese dataset, we achieved 8.65% equal error rate (EER) that means 1.2% relative improvement compared to the best-reported result. Furthermore, on the GPDSsyntheticOnLineOffLineSignature dataset, we achieved average EERs of 0.13%, 0.12%, 0.21% and 0.25%, respectively, for 150, 300, 1000 and 2000 test subjects which indicate improvement in relative EER on the best-reported result by 95.67%, 95.26%, 92.9% and 91.52%, respectively. Apart from the accuracy gain, because of the nature of our proposed framework which is based on neural networks and consequently is as simple as some consecutive matrix multiplications, it has less computational cost than conventional methods such as Dynamic Time Warping and could be used concurrently on devices such as Graphics Processing Unit and Tensor Processing Unit.

SolarGest: Ubiquitous and Battery-free Gesture Recognition using Solar Cells

Abstract: We design a system, SolarGest, which can recognize hand gestures near a solar-powered device by analyzing the patterns of the photocurrent. SolarGest is based on the observation that each gesture interferes with incident light rays on the solar panel in a unique way, leaving its distinguishable signature in harvested photocurrent. Using solar energy harvesting laws, we develop a model to optimize design and usage of SolarGest. To further improve the robustness of SolarGest under non-deterministic operating conditions, we combine dynamic time warping with Z-score transformation in a signal processing pipeline to pre-process each gesture waveform before it is analyzed for classification. We evaluate SolarGest with both conventional opaque solar cells as well as emerging see-through transparent cells. Our experiments with 6,960 gesture samples for 6 different gestures reveal that even with transparent cells, SolarGest can detect 96% of the gestures while consuming 44% less power compared to light sensor based systems.

Wi-Multi: A Three-Phase System for Multiple Human Activity Recognition With Commercial WiFi Devices

Abstract: Channel state information-based activity recognition has gathered immense attention over recent years. Many existing works achieved desirable performance in various applications, including healthcare, security, and Internet of Things, with different machine learning algorithms. However, they usually fail to consider the availability of enough samples to be trained. Besides, many applications only focus on the scenario where only single subject presents. To address these challenges, in this paper, we propose a three-phase system Wi-multi that targets at recognizing multiple human activities in a wireless environment. Different system phases are applied according to the size of available collected samples. Specifically, distance-based classification using dynamic time warping is applied when there are few samples in the profile. Then, support vector machine is employed when representative features can be extracted from training samples. Lastly, recurrent neural networks is exploited when a large number of samples are available. Extensive experiments results show that Wi-multi achieves an accuracy of 96.1% on average. It is also able to achieve a desirable tradeoff between accuracy and efficiency in different phases.

GaitLock: Protect Virtual and Augmented Reality Headsets Using Gait

Abstract: With the fast penetration of commercial Virtual Reality (VR) and Augmented Reality (AR) systems into our daily life, the security issues of those devices have attracted significant interests from both academia and industry. Modern VR/AR systems typically use head-mounted devices (i.e., headsets) to interact with users, and often store private user data, e.g., social network accounts, online transactions or even payment information. This poses significant security threats, since in practice the headset can be potentially obtained and accessed by unauthenticated parties, e.g., identity thieves, and thus cause catastrophic breach. In this paper, we propose a novel GaitLock system, which can reliably authenticate users using their gait signatures. Our system doesn't require extra hardware, e.g., fingerprint sensors or retina scanners, but only uses the on-board inertial measurement units (IMUs) equipped in almost all mainstream VR/AR headsets to authenticate the legitimate users from intruders, by simply asking them to walk a few steps. To achieve that, we propose a new gait recognition model Dynamic-SRC, which combines the strength of Dynamic Time Warping (DTW) and Sparse Representation Classifier (SRC), to extract unique gait patterns from the inertial signals during walking. We implement GaitLock on Google Glass (a typical AR headset), and extensive experiments show that GaitLock outperforms the state-of-the-art systems significantly in recognition accuracy ($>$>98 percent success in 5 steps), and is able to run in-situ on the resource-constrained VR/AR headsets without incurring high energy cost.

General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping

Abstract: In instruction conditioned navigation, agents interpret natural language and their surroundings to navigate through an environment. Datasets for studying this task typically contain pairs of these instructions and reference trajectories. Yet, most evaluation metrics used thus far fail to properly account for the latter, relying instead on insufficient similarity comparisons. We address fundamental flaws in previously used metrics and show how Dynamic Time Warping (DTW), a long known method of measuring similarity between two time series, can be used for evaluation of navigation agents. For such, we define the normalized Dynamic Time Warping (nDTW) metric, that softly penalizes deviations from the reference path, is naturally sensitive to the order of the nodes composing each path, is suited for both continuous and graph-based evaluations, and can be efficiently calculated. Further, we define SDTW, which constrains nDTW to only successful paths. We collect human similarity judgments for simulated paths and find nDTW correlates better with human rankings than all other metrics. We also demonstrate that using nDTW as a reward signal for Reinforcement Learning navigation agents improves their performance on both the Room-to-Room (R2R) and Room-for-Room (R4R) datasets. The R4R results in particular highlight the superiority of SDTW over previous success-constrained metrics.

Practical Approaches Toward Deep-Learning-Based Cross-Device Power Side-Channel Attack

Abstract: Power side-channel analysis (SCA) has been of immense interest to most embedded designers to evaluate the physical security of the system. This work presents profiling-based cross-device power SCA attacks using deep-learning techniques on 8-bit AVR microcontroller devices running AES-128. First, we show the practical issues that arise in these profiling-based cross-device attacks due to significant device-to-device variations. Second, we show that utilizing principal component analysis (PCA)-based preprocessing and multidevice training, a multilayer perceptron (MLP)-based 256-class classifier can achieve an average accuracy of 99.43% in recovering the first keybyte from all the 30 devices in our data set, even in the presence of significant interdevice variations. Results show that the designed MLP with PCA-based preprocessing outperforms a convolutional neural network (CNN) with four-device training by ~20% in terms of the average test accuracy of cross-device attack for the aligned traces captured using the ChipWhisperer hardware. Finally, to extend the practicality of these cross-device attacks, another preprocessing step, namely, dynamic time warping (DTW) has been utilized to remove any misalignment among the traces, before performing PCA. DTW along with PCA followed by the 256-class MLP classifier provides ≥10.97% higher accuracy than the CNN-based approach for cross-device attack even in the presence of up to 50 time-sample misalignments between the traces.

Temporal Transformer Networks: Joint Learning of Invariant and Discriminative Time Warping

Abstract: Many time-series classification problems involve developing metrics that are invariant to temporal misalignment. In human activity analysis, temporal misalignment arises due to various reasons including differing initial phase, sensor sampling rates, and elastic time-warps due to subject-specific biomechanics. Past work in this area has only looked at reducing intra-class variability by elastic temporal alignment. In this paper, we propose a hybrid model-based and data-driven approach to learn warping functions that not just reduce intra-class variability, but also increase inter-class separation. We call this a temporal transformer network (TTN). TTN is an interpretable differentiable module, which can be easily integrated at the front end of a classification network. The module is capable of reducing intra-class variance by generating input-dependent warping functions which lead to rate-robust representations. At the same time, it increases inter-class variance by learning warping functions that are more discriminative. We show improvements over strong baselines in 3D action recognition on challenging datasets using the proposed framework. The improvements are especially pronounced when training sets are smaller.

Template Matching Using Time-Series Averaging and DTW With Dependent Warping for Online Signature Verification

Abstract: Online signature verification has been widely applied in biometrics and forensics. Due to the recent demand on high-speed systems in this era of big data, to simultaneously improve its performance and calculation complexity, this study focuses on a single-template strategy that uses dynamic time warping (DTW) with dependent warping for online signature verification, and attempts to construct a novel time-series averaging method called Euclidean barycenter-based DTW barycenter averaging (EB-DBA). Specifically, this study proposes a single-template strategy using a mean template created by the EB-DBA to achieve higher performance at lower calculation complexity for online signature verification. The method's discriminative power is enhanced upon the exploration of two DTW warping types, where it is found that the DTW with dependent warping exhibits better performance. The popular MCYT-100 dataset is utilized in the experiments, which confirms the effectiveness of the proposed method in simultaneously achieving lower error rate and lower calculation complexity, for online signature verification.

Data Augmentation with Suboptimal Warping for Time-Series Classification

Abstract: In this paper, a novel data augmentation method for time-series classification is proposed. In the introduced method, a new time-series is obtained in warped space between suboptimally aligned input examples of different lengths. Specifically, the alignment is carried out constraining the warping path and reducing its flexibility. It is shown that the resultant synthetic time-series can form new class boundaries and enrich the training dataset. In this work, the comparative evaluation of the proposed augmentation method against related techniques on representative multivariate time-series datasets is presented. The performance of methods is examined using the nearest neighbor classifier with the dynamic time warping (NN-DTW), LogDet divergence-based metric learning with triplet constraints (LDMLT), and the recently introduced time-series cluster kernel (NN-TCK). The impact of the augmentation on the classification performance is investigated, taking into account entire datasets and cases with a small number of training examples. The extensive evaluation reveals that the introduced method outperforms related augmentation algorithms in terms of the obtained classification accuracy.

DTWNet: a Dynamic Time Warping Network

Abstract: Dynamic Time Warping (DTW) is widely used as a similarity measure in various domains. Due to its invariance against warping in the time axis, DTW provides more meaningful discrepancy measurements between two signals than other dis- tance measures. In this paper, we propose a novel component in an artificial neural network. In contrast to the previous successful usage of DTW as a loss function, the proposed framework leverages DTW to obtain a better feature extraction. For the first time, the DTW loss is theoretically analyzed, and a stochastic backpropogation scheme is proposed to improve the accuracy and efficiency of the DTW learning. We also demonstrate that the proposed framework can be used as a data analysis tool to perform data decomposition.

A novel hidden Markov model-based adaptive dynamic time warping (HMDTW) gait analysis for identifying physically challenged persons

Abstract: Internet of things plays vital role in real-time applications, and the research thrust towards implementing IoT in gait analysis increases day by day in order to obtain efficient gait recognition mechanism. IoT in gait analysis is used to monitor and communicate the observing gait, and also to transfer data to others is the current trend which is available. This research work provides an efficient gait recognition system with IoT using dynamic time wrapping and naive bays classifier as combination to obtain hybrid model. The objective of this research is identifying the patients or persons with walking disabilities in a crowded area and providing suitable alerts to them by monitoring the walking styles. So that the possibility of getting injured is avoided and the information related to the persons also alerted through IoT module. Also, IoT module is used to collect information from the sensors used in persons accessories and other places. Twenty-five males and 10 females are subjected to examine the proposed model in different locations and achieved the overall accuracy percentage of 92.15%.

Sensor-based machine learning for workflow detection and as key to detect expert level in laparoscopic suturing and knot-tying

Abstract: The most common way of assessing surgical performance is by expert raters to view a surgical task and rate a trainee’s performance. However, there is huge potential for automated skill assessment and workflow analysis using modern technology. The aim of the present study was to evaluate machine learning (ML) algorithms using the data of a Myo armband as a sensor device for skills level assessment and phase detection in laparoscopic training. Participants of three experience levels in laparoscopy performed a suturing and knot tying task on silicon models. Experts rated performance using Objective Structured Assessment of Surgical Skills (OSATS). Participants wore Myo armbands (Thalmic Labs™, Ontario, Canada) to record acceleration, angular velocity, orientation, and Euler orientation. ML algorithms (decision forest, neural networks, boosted decision tree) were compared for skill level assessment and phase detection. 28 participants (8 beginner, 10 intermediate, 10 expert) were included, and 99 knots were available for analysis. A neural network regression model had the lowest mean absolute error in predicting OSATS score (3.7 ± 0.6 points, r2 = 0.03 ± 0.81; OSATS min.-max.: 4–37 points). An ensemble of binary-class neural networks yielded the highest accuracy in predicting skill level (beginners: 82.2% correctly identified, intermediate: 3.0%, experts: 79.5%) whereas standard statistical analysis failed to discriminate between skill levels. Phase detection on raw data showed the best results with a multi-class decision jungle (average 16% correctly identified), but improved to 43% average accuracy with two-class boosted decision trees after Dynamic time warping (DTW) application. Modern machine learning algorithms aid in interpreting complex surgical motion data, even when standard analysis fails. Dynamic time warping offers the potential to process and compare surgical motion data in order to allow automated surgical workflow detection. However, further research is needed to interpret and standardize available data and improve sensor accuracy.

Dynamic time warping in classification and selection of motion capture data

Abstract: The paper is a comprehensive study on classification of motion capture data on the basis of dynamic time warping (DTW) transform. It presents both theoretical description of all applied and newly proposed methods and experimentally obtained results on real dataset of human gait with 436 samples of 30 males. The recognition is carried out by the classical DTW nearest neighbors classifier and introduced DTW minimum distance scheme. Class prototypes are determined on the basis of DTW alignment and chosen methods of averaging rotations represented by Euler angles and unit quaternions. In the basic classification approach the whole pose configuration space is taken into account. The influence of different rotation distance functions operating on Euler angles and unit quaternions, on an obtained accuracy of recognition is investigated. What is more, a differential filtering in time domain which approximates angular velocities and accelerations of subsequent joints is utilized. Because in the case of unit quaternions representing rotations classical subtraction is unworkable, the differential filtering based on a product with a conjugated quaternion is applied. The main contribution of the paper is also related to the proposed and successfully validated approach of an exploration of pose configuration space. It selects the most discriminative joints of a skeleton model in considered classification problem in a binary or fuzzy way. The introduced approach utilizes hill climbing and genetic search strategies as well as DTW transform based evaluation. The selection makes the recognition more efficient and reduces pose signatures.

Multimodal One-shot Learning of Speech and Images

Abstract: Imagine a robot is shown new concepts visually together with spoken tags, e.g. "milk", "eggs", "butter". After seeing one paired audiovisual example per class, it is shown a new set of unseen instances of these objects, and asked to pick the "milk". Without receiving any hard labels, could it learn to match the new continuous speech input to the correct visual instance? Although unimodal one-shot learning has been studied, where one labelled example in a single modality is given per class, this example motivates multimodal one-shot learning. Our main contribution is to formally define this task, and to propose several baseline and advanced models. We use a dataset of paired spoken and visual digits to specifically investigate recent advances in Siamese convolutional neural networks. Our best Siamese model achieves twice the accuracy of a nearest neighbour model using pixel-distance over images and dynamic time warping over speech in 11-way cross-modal matching.