Showing papers by "Paul Sajda published in 2018"

Compact Convolutional Neural Networks for Classification of Asynchronous Steady-state Visual Evoked Potentials

Abstract: Steady-State Visual Evoked Potentials (SSVEPs) are neural oscillations from the parietal and occipital regions of the brain that are evoked from flickering visual stimuli. SSVEPs are robust signals measurable in the electroencephalogram (EEG) and are commonly used in brain-computer interfaces (BCIs). However, methods for high-accuracy decoding of SSVEPs usually require hand-crafted approaches that leverage domain-specific knowledge of the stimulus signals, such as specific temporal frequencies in the visual stimuli and their relative spatial arrangement. When this knowledge is unavailable, such as when SSVEP signals are acquired asynchronously, such approaches tend to fail. In this paper, we show how a compact convolutional neural network (Compact-CNN), which only requires raw EEG signals for automatic feature extraction, can be used to decode signals from a 12-class SSVEP dataset without the need for any domain-specific knowledge or calibration data. We report across subject mean accuracy of approximately 80% (chance being 8.3%) and show this is substantially better than current state-of-the-art hand-crafted approaches using canonical correlation analysis (CCA) and Combined-CCA. Furthermore, we analyze our Compact-CNN to examine the underlying feature representation, discovering that the deep learner extracts additional phase and amplitude related features associated with the structure of the dataset. We discuss how our Compact-CNN shows promise for BCI applications that allow users to freely gaze/attend to any stimulus at any time (e.g., asynchronous BCI) as well as provides a method for analyzing SSVEP signals in a way that might augment our understanding about the basic processing in the visual cortex.

Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials

Abstract: Objective Steady-state visual evoked potentials (SSVEPs) are neural oscillations from the parietal and occipital regions of the brain that are evoked from flickering visual stimuli. SSVEPs are robust signals measurable in the electroencephalogram (EEG) and are commonly used in brain-computer interfaces (BCIs). However, methods for high-accuracy decoding of SSVEPs usually require hand-crafted approaches that leverage domain-specific knowledge of the stimulus signals, such as specific temporal frequencies in the visual stimuli and their relative spatial arrangement. When this knowledge is unavailable, such as when SSVEP signals are acquired asynchronously, such approaches tend to fail. Approach In this paper, we show how a compact convolutional neural network (Compact-CNN), which only requires raw EEG signals for automatic feature extraction, can be used to decode signals from a 12-class SSVEP dataset without the need for user-specific calibration. Main results The Compact-CNN demonstrates across subject mean accuracy of approximately 80%, out-performing current state-of-the-art, hand-crafted approaches using canonical correlation analysis (CCA) and Combined-CCA. Furthermore, the Compact-CNN approach can reveal the underlying feature representation, revealing that the deep learner extracts additional phase- and amplitude-related features associated with the structure of the dataset. Significance We discuss how our Compact-CNN shows promise for BCI applications that allow users to freely gaze/attend to any stimulus at any time (e.g. asynchronous BCI) as well as provides a method for analyzing SSVEP signals in a way that might augment our understanding about the basic processing in the visual cortex.

Relating Deep Neural Network Representations to EEG-fMRI Spatiotemporal Dynamics in a Perceptual Decision-Making Task

Abstract: The hierarchical architecture of deep convolutional neural networks (CNN) resembles the multi-level processing stages of the human visual system during object recognition Converging evidence suggests that this hierarchical organization is key to the CNN achieving human-level performance in object categorization [22] In this paper, we leverage the hierarchical organization of the CNN to investigate the spatiotemporal dynamics of rapid visual processing in the human brain Specifically we focus on perceptual decisions associated with different levels of visual ambiguity Using simultaneous EEG-fMRI, we demonstrate the temporal and spatial hierarchical correspondences between the multi-stage processing in CNN and the activity observed in the EEG and fMRI The hierarchical correspondence suggests a processing pathway during rapid visual decision-making that involves the interplay between sensory regions, the default mode network (DMN) and the frontal-parietal control network (FPCN)

Regulation of arousal via on-line neurofeedback improves human performance in a demanding sensory-motor task

Abstract: Our state of arousal can significantly affect our ability to make optimal decisions, judgments, and actions in real-world dynamic environments. The Yerkes-Dodson law, which posits an inverse-U relationship between arousal and task performance, suggests that there is a state of arousal that is optimal for behavioral performance in a given task. Here we show that we can use on-line neurofeedback to shift an individual's arousal toward this optimal state. Specifically, we use a brain computer interface (BCI) that uses information in the electroencephalogram (EEG) to generate a neurofeedback signal that dynamically adjusts an individual's arousal state when they are engaged in a boundary avoidance task (BAT). The BAT is a demanding sensory-motor task paradigm that we implement as an aerial navigation task in virtual reality (VR), and which creates cognitive conditions that escalate arousal and quickly results in task failure - e.g. missing or crashing into the boundary. We demonstrate that task performance, measured as time and distance over which the subject can navigate before failure, is significantly increased when veridical neurofeedback is provided. Simultaneous measurements of pupil dilation and heart rate variability show that the neurofeedback indeed reduces arousal. Our work is the first demonstration of a BCI system that uses on-line neurofeedback to shift arousal state and increase task performance in accordance with the Yerkes-Dodson law.

EEG-based Endogenous Online Co-Adaptive Brain-Computer Interfaces: Strategy for Success?

Abstract: A Brain-Computer Interface (BCI) translates patterns of brain signals such as the electroencephalogram (EEG) into messages for communication and control. In the case of endogenous systems the reliable detection of induced patterns is more challenging than the detection of the more stable and stereotypical evoked responses. In the former case specific mental activities such as motor imagery are used to encode different messages. In the latter case users have to attend to sensory stimuli to evoke a characteristic response. Indeed, a large number of users who try to control endogenous BCIs do not reach sufficient level of accuracy. This fact is also known as BCI “inefficiency” or “illiteracy”. In this paper we discuss and make some conjectures, based on our knowledge and experience in BCI, on whether or not online co-adaptation of human and machine can be the solution to overcome this challenge. We point out some ingredients that might be necessary for the system to be reliable and allow the users to attain sufficient control.

Spatio-temporally specific transcranial magnetic stimulation to test the locus of perceptual decision making in the human brain

Abstract: Previous research modeling EEG, fMRI and behavioral data has identified three spatially distributed brain networks that activate in temporal sequence, and are thought to enable perceptual decision-making during face-versus-car categorization. These studies have linked late activation (>300ms post stimulus onset) in the lateral occipital cortex (LOC) to object discrimination processes. We applied paired-pulse transcranial magnetic stimulation (ppTMS) to LOC at different temporal latencies with the specific prediction, based on these studies, that ppTMS beginning at 400ms after stimulus onset would slow reaction time (RT) performance. Thirteen healthy adults performed a two-alternative forced choice task selecting whether a car or face was present on each trial amidst visual noise pre-titrated to approximate 79% accuracy. ppTMS, with pulses separated by 50ms, was applied at one of five stimulus onset asynchronies: -200, 200, 400, 450, or 500ms, and a sixth no-stimulation condition. As predicted, TMS at 400ms resulted in significant slowing of RTs, providing causal evidence in support of LOC contribution to perceptual decision processing. In addition, TMS delivered at -200ms resulted in faster RTs, indicating early stimulation may result in performance enhancement. These findings build upon correlational EEG and fMRI observations and demonstrate the use of TMS in predictive validation of psychophysiological models.

Information-Theoretic Characterization of the Neural Mechanisms of Active Multisensory Decision Making

Abstract: The signals delivered by different sensory modalities provide us with complementary information about the environment. A key component of interacting with the world is how to direct ones’ sensors so as to extract task-relevant information in order to optimize subsequent perceptual decisions. This process is often referred to as active sensing. Importantly, the processing of multisensory information acquired actively from multiple sensory modalities requires the interaction of multiple brain areas over time. Here we investigated the neural underpinnings of active visual-haptic integration during performance of a two-alternative forced choice (2AFC) reaction time (RT) task. We asked human subjects to discriminate the amplitude of two texture stimuli (a) using only visual (V) information, (b) using only haptic (H) information and (c) combining the two sensory cues (VH), while electroencephalograms (EEG) were recorded. To quantify multivariate interactions between EEG signals and active sensory experience in the three sensory conditions, we employed a novel information-theoretic methodology. This approach provides a principled way to quantify the contribution of each one of the sensory modalities to the perception of the stimulus and assess whether the respective neural representations may interact to form a percept of the stimulus and ultimately drive perceptual decisions. Application of this method to our data identified (a) an EEG component (comprising frontal and occipital electrodes) carrying behavioral information that is common to the two sensory inputs and (b) another EEG component (mainly motor) reflecting a synergistic representational interaction between the two sensory inputs. We suggest that the proposed approach can be used to elucidate the neural mechanisms underlying cross-modal interactions in active multisensory processing and decision-making.

Brain–Computer Interfaces for Mediating Interaction in Virtual and Augmented Reality

Abstract: Virtual and augmented reality (VR/AR) are immersive and potentially multimodal sensory experiences that augment or completely replace real-world sensory input with artificial content. Outside commercial applications in entertainment, VR and AR can be used to create controlled, yet ecologically valid experimental paradigms for research in cognitive science or related fields. Here, we review how brain–computer interfaces (BCIs) are being integrated with VR/AR to rehabilitate and assist the injured and disabled, improve interaction between human and computer, and provide us more insight into how our brains process and evaluate complex environments and events. We also describe a concrete example architecture for conducting BCI/VR experiments and conclude our review with a discussion on limitations and potential future developments on BCI-based interaction with VR and AR.

Correlations of Neural Activity with Behavioral Kinematics during Active Tactile Decision Making

Abstract: Most real-world decisions rely on active sensing, a dynamic process for directing our sensors (e.g. eyes or fingers) across a stimulus in order to reduce uncertainty and maximize information gain. Though ecologically pervasive, relatively limited work has focused on identifying neural correlates of the active sensing process. In tactile perception, we often make decisions about an object or surface by actively exploring its shape and texture. Here we investigate the neural mechanisms of active tactile sensing by simultaneously measuring electroencephalography (EEG) and finger kinematics while subjects interrogated a haptic surface to make perceptual judgements. We hypothesized that one's sensorimotor behavior provides a view into the cognitive processes leading to decision formation, and the neural correlates of these processes would be detectable by relating kinematics to neural activity. Using an adaptation of canonical correlation analysis (CCA), we regressed the EEG onto kinematics and found three distinct, task-related EEG components that localized to right-lateralized occipital cortex (LOC), middle frontal gyrus (MFG), and supplementary motor area (SMA), respectively. To probe the functional role of these components, we fit their single-trial activity to behavior using a hierarchical drift diffusion model (HDDM), revealing that the LOC modulated the encoding of the tactile stimulus whereas the MFG predicted the rate of information integration towards a choice. This study provides direct evidence that, how we explore the stimulus yields insight into how our brain is forming a decision and uncovers the neural correlates of distinct sensory encoding and evidence accumulation processes during active tactile sensing.

Decomposing spatial conflict BOLD activation using a drift-diffusion model framework

Abstract: Drift-diffusion based models (DDM) have been recently adapted for analysis of behavior in spatial conflict tasks. However such DDM extensions are typically difficult to fit and compare because analytical solutions do not exist. We use a numerical method to estimate the likelihood function for fitting a Simon effect DDM to individual subject data, and use these fits to interpret blood-oxygenlevel dependent (BOLD) responses. We find regions of BOLD activation that would be difficult to observe with methods that are not model based.

Parameterized expertise: Evidence for gradations of musical expertise from electroencephalographic monitoring of auditory decision-making

Abstract: A musician9s nervous system is thought to specialize to their mode of music production. For instance, the pianist9s control of hands and arms develops to facilitate greater dexterity at the keyboard, while the cellist develops control to play notes using both the fret board and/or bow. Our previous work, employing an anomalous musical event (AME) detection task, identified neural and behavioral correlates that differentiated between a specific class of musicians, cellists, and those without professional musical training and expertise. Here we investigate a fine-grain differentiation between musicians having different modes of musical production, specifically in terms of how these differences are manifested in the neural correlates identified in the AME task. We show, using electroencephalography (EEG), that both event related potentials (ERPs) and single-trial analysis of the EEG can grade musical expertise by mode of sound production. Important is that these fine-grained EEG correlates are observable absent any motor response or active music production by the individuals. We find evidence that these grades of expertise are mediated by different sensory-motor interactions emblematic of the sound production mode. More broadly, our results show that neural markers can both define types of musical expertise and decompose their source components when behavioral differences are either minute or indistinguishable.