Data quality is essential to the validity of research results and to the quality of gaze interaction. We argue that the lack of standard measures for eye data quality makes several aspects of manufacturing and using eye trackers, as well as researching eye movements and vision, more difficult than necessary. Uncertainty regarding the comparability of research results is a considerable impediment to progress in the field. In this paper, we illustrate why data quality matters and review previous work on how eye data quality has been measured and reported. The goal is to achieve a common understanding of what data quality is and how it can be defined, measured, evaluated, and reported.

Eye tracker data quality: what it is and how to measure it

In an effort toward standardization, this paper evaluates the performance of five eye-movement classification algorithms in terms of their assessment of oculomotor fixation and saccadic behavior. The results indicate that performance of these five commonly used algorithms vary dramatically, even in the case of a simple stimulus-evoked task using a single, common threshold value. The important contributions of this paper are: evaluation and comparison of performance of five algorithms to classify specific oculomotor behavior; introduction and comparison of new standardized scores to provide more reliable classification performance; logic for a reasonable threshold-value selection for any eye-movement classification algorithm based on the standardized scores; and logic for establishing a criterion-based baseline for performance comparison between any eye-movement classification algorithms. Proposed techniques enable efficient and objective clinical applications providing means to assure meaningful automated eye-movement classification.

Standardization of Automated Analyses of Oculomotor Fixation and Saccadic Behaviors

Almost all eye-movement researchers use algorithms to parse raw data and detect distinct types of eye movement events, such as fixations, saccades, and pursuit, and then base their results on these. Surprisingly, these algorithms are rarely evaluated. We evaluated the classifications of ten eye-movement event detection algorithms, on data from an SMI HiSpeed 1250 system, and compared them to manual ratings of two human experts. The evaluation focused on fixations, saccades, and post-saccadic oscillations. The evaluation used both event duration parameters, and sample-by-sample comparisons to rank the algorithms. The resulting event durations varied substantially as a function of what algorithm was used. This evaluation differed from previous evaluations by considering a relatively large set of algorithms, multiple events, and data from both static and dynamic stimuli. The main conclusion is that current detectors of only fixations and saccades work reasonably well for static stimuli, but barely better than chance for dynamic stimuli. Differing results across evaluation methods make it difficult to select one winner for fixation detection. For saccade detection, however, the algorithm by Larsson, Nystrom and Stridh (IEEE Transaction on Biomedical Engineering, 60(9):2484–2493,2013) outperforms all algorithms in data from both static and dynamic stimuli. The data also show how improperly selected algorithms applied to dynamic data misestimate fixation and saccade properties.

/pdf/one-algorithm-to-rule-them-all-an-evaluation-and-discussion-d2aw7zz5h9.pdf

One algorithm to rule them all? An evaluation and discussion of ten eye movement event-detection algorithms

Wearable eye tracking for mental health monitoring

This paper presents an objective evaluation of various eye movement-based biometric features and their ability to accurately and precisely distinguish unique individuals. Eye movements are uniquely counterfeit resistant due to the complex neurological interactions and the extraocular muscle properties involved in their generation. Considered biometric candidates cover a number of basic eye movements and their aggregated scanpath characteristics, including: fixation count, average fixation duration, average saccade amplitudes, average saccade velocities, average saccade peak velocities, the velocity waveform, scanpath length, scanpath area, regions of interest, scanpath inflections, the amplitude-duration relationship, the main sequence relationship, and the pairwise distance between fixations. As well, an information fusion method for combining these metrics into a single identification algorithm is presented. With limited testing this method was able to identify subjects with an equal error rate of 27%. These results indicate that scanpath-based biometric identification holds promise as a behavioral biometric technique.

Biometric identification via eye movement scanpaths in reading

This paper presents a set of qualitative and quantitative scores designed to assess performance of any eye movement classification algorithm. The scores are designed to provide a foundation for the eye tracking researchers to communicate about the performance validity of various eye movement classification algorithms. The paper concentrates on the five algorithms in particular: Velocity Threshold Identification (I-VT), Dispersion Threshold Identification (I-DT), Minimum Spanning Tree Identification (MST), Hidden Markov Model Identification (I-HMM) and Kalman Filter Identification (I-KF). The paper presents an evaluation of the classification performance of each algorithm in the case when values of the input parameters are varied. Advantages provided by the new scores are discussed. Discussion on what is the "best" classification algorithm is provided for several applications. General recommendations for the selection of the input parameters for each algorithm are provided.

/pdf/qualitative-and-quantitative-scoring-and-evaluation-of-the-4qqe2t50z9.pdf

Qualitative and quantitative scoring and evaluation of the eye movement classification algorithms

This paper evaluates the input performance capabilities of Velocity Threshold (I-VT) and Kalman Filter (I-KF) eye movement detection models when employed for eye-gaze-guided interface control. I-VT is a common eye movement identification model employed by the eye tracking community, but it is neither robust nor capable of handling high levels of noise present in the eye position data. Previous research implies that use of a Kalman filter reduces the noise in the eye movement signal and predicts the signal during brief eye movement failures, but the actual performance of I-KF was never evaluated. We evaluated the performance of I-VT and I-KF models using guidelines for ISO 9241 Part 9 standard, which is designed for evaluation of non keyboard/mouse input devices with emphasis on performance, comfort, and effort. Two applications were implemented for the experiment: 1) an accuracy test 2) a photo viewing application specifically designed for eye-gaze-guided control. Twenty-one subjects participated in the evaluation of both models completing a series of tasks. The results indicates that I-KF allowed participants to complete more tasks with shorter completion time while providing higher general comfort, accuracy and operation speeds with easier target selection than the I-VT model. We feel that these results are especially important to the engineers of new assistive technologies and interfaces that employ eye-tracking technology in their design.

/pdf/input-evaluation-of-an-eye-gaze-guided-interface-kalman-4cdc4t0e3m.pdf

Input evaluation of an eye-gaze-guided interface: kalman filter vs. velocity threshold eye movement identification

In this paper, we introduce and evaluate a new Instantaneous Saccade (IS) selection scheme for eye gaze driven interfaces where the speed of the target selection is of utmost importance. In the IS selection scheme, target selection occurs at the start (onset) of a saccade requiring only constant amount of time to be completed. The IS performance is compared to the conventional Dwell Time (DT) selection scheme where target selection is triggered when a user fixates on an object for a certain amount of time. The IS method is also compared to the Saccade Offset (SO) selection scheme where target selection occurs at the end of a saccade. All three schemes were evaluated in terms of task completion time and the throughput of input performance in horizontal target selection task by six subjects. Results show that the Instantaneous Saccade selection was 57% faster than the DT selection to complete a task. In terms of throughput comparison, the throughput of the IS selection is 1.9 times greater than the throughput of DT selection. We hypothesize that Instantaneous Saccade selection will be beneficial in gaming environments that require fast very interaction speeds.

Instantaneous saccade driven eye gaze interaction

This paper introduces a Real Time Eye Movement Identification (REMI) protocol designed to address challenges related to the implementation of the eye-gaze guided computer interfaces. The REMI protocol provides the framework for 1) eye position data processing such as noise removal, smoothing, prediction and handling of invalid positional samples 2) real time eye movement identification into the basic eye movement types 3) mapping of the classified eye movement data to interface actions such as object selection.

/pdf/real-time-eye-movement-identification-protocol-2bor9v40q4.pdf

Sandeep Munikrishne Gowda

Papers

Standardization of Automated Analyses of Oculomotor Fixation and Saccadic Behaviors

Qualitative and quantitative scoring and evaluation of the eye movement classification algorithms

Input evaluation of an eye-gaze-guided interface: kalman filter vs. velocity threshold eye movement identification

Instantaneous saccade driven eye gaze interaction

Real time eye movement identification protocol