Recent Advances in Brain-Computer Interfaces
(Invited Paper)
Ulrich Hoffmann, Jean-Marc Vesin, Touradj Ebrahimi
Signal Processing Institute
Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Switzerland
Email: {ulrich.hoffmann, jean-marc.vesin, touradj.ebrahimi}@epfl.ch
Abstract— A brain-computer interface (BCI) is a communi-
cation system that translates brain activity into commands for
a computer or other devices. In other words, a BCI allows
users to act on their environment by using only brain activity,
without using peripheral nerves and muscles. The major goal of
BCI research is to develop systems that allow disabled users to
communicate with other persons, to control artificial limbs, or
to control their environment. An alternative application area for
brain-computer interfaces (BCIs) lies in the field of multimedia
communication. To develop systems for usage in the field of
assistive technology or multimedia communication, many aspects
of BCI systems are currently being investigated. Research areas
include evaluation of invasive and noninvasive technologies to
measure brain activity, evaluation of control signals (i.e. patterns
of brain activity that can be used for communication), develop-
ment of algorithms for translation of brain signals into computer
commands, and the development of new BCI applications. In this
paper we give an introduction to some of the aspects of BCI
research mentioned above, present a concrete example of a BCI
system, and highlight recent developments and open problems.
I. INTRODUCTION
The ability to communicate with other persons, be it through
speech, gesturing, or writing, is one of the main factors making
the life of any human being enjoyable. Communication is at
the basis of human development, makes it possible to express
ideas, desires, and feelings, and on a more ordinary level
simply allows to cope with daily life.
Individuals suffering from the so-called locked-in syndrome
do not have the above mentioned communication possibilities.
The locked-in syndrome is a condition in which patients
are fully conscious and aware of what is happening in their
environment but are not able to communicate or move. In
fact, the locked-in syndrome is caused by a nearly total
loss of control over the voluntary muscles. A disease that is
known to lead to the locked-in syndrome is amyotrophic lateral
sclerosis (ALS), also known as Lou Gehrig’s disease. ALS is
a progressive, neurodegenerative disease and is characterized
by the death of motor neurons which in turn leads to the loss
of control over voluntary muscles. Besides ALS also multiple
sclerosis, stroke or other cerebrovascular incidents leading to
the infarction or degeneration of parts of the brain can cause
the locked-in syndrome. Clearly, the quality of live of persons
affected by the locked-in syndrome is strongly diminished by
the lack of possibilities to communicate with other persons
and by the complete loss of autonomy.
A promising means to give back basic communication
abilities and a small degree of autonomy to locked-in persons
are BCIs. The idea underlying BCIs is to measure electric,
magnetic, or other physical manifestations of brain activity
and to translate these into commands for a computer or other
devices (see Fig. 1).
From a different angle, BCIs can also be seen as a new and
exciting means of communication that could be used as well
by persons not suffering from disabilities. For example, in the
field of multimedia communication and human-computer inter-
action, BCIs could possibly be used as an additional modality,
together with more traditional modalities, such as the auditive
and visual modalities [1]. Multimodal communication with
the help of a BCI would help to increase the communication
bandwidth between man and machine. Beyond communica-
tion, other applications of BCI involving multimedia can also
be envisioned. For example one can imagine (multiplayer)
games in which BCIs are used for control. Another interesting
application area might be the visualization, or sonification,
i.e. the transformation into sound, of brain activity.
Independently of the application in the fields of assistive
technology or multimedia, the aim of this paper is to give an
introduction to the field of BCI research. In the first part of the
paper (Sections II, III, IV) we review neurophysiologic signals
that can be used in BCIs, signal processing and machine
learning methods for BCIs, and applications for BCIs. In the
second part of the paper (Section V) a concrete state-of-the-art
BCI system is briefly described. Finally, in the third part of
the paper (Section VI) some open problems in BCI research
are mentioned.
Note that the material presented here is strongly biased
towards BCI systems using the electroencephalogram (EEG)
as measurement technology. Other reviews can be found in
[2], [3] and [4].
II. N
EUROPHYSIOLOGIC SIGNALS
To control a BCI, users have to acquire conscious con-
trol over their brain activity. Two fundamentally different
approaches exist to achieve this.
In the first approach, subjects perceive a set of stimuli dis-
played by the BCI system and can control their brain activity
by focusing onto one specific stimulus. The changes in neuro-
physiologic signals resulting from perception and processing
of stimuli are termed event-related potentials (ERPs) and are
discussed together with the corresponding BCI paradigms in
subsection II-A.
Signal Acquisition
Signal Processing
Classification
Application
Feedback
Fig. 1. Building blocks of a BCI. A subject performs a specific cognitive task or concentrates on a specific stimulus. Brain signals are acquired and then
processed with signal processing and classification algorithms. The outcome of the classification is fed into an application, for example a spelling device. The
application generates feedback to inform the subject about the outcome of classification.
In the second approach, users control their brain activity
by concentrating on a specific mental task. For example
imagination of hand movement can be used to modify activity
in the motor cortex. In this approach feedback signals are often
used to let subjects learn the production of easily detectable
patterns of neurophysiologic signals. The types of signals
resulting from concentration on mental tasks together with the
corresponding BCI paradigms are described in subsections II-
B, II-C, and II-D.
A. Event-Related Potentials
ERPs are stereotyped, spatio-temporal patterns of brain
activity, occurring time-locked to an event, for example after
presentation of a stimulus, before execution of a movement,
or after the detection of a novel stimulus.
An example for an ERP that is often used in BCIs is the
so-called P300. The P300 is a positive deflection in the EEG,
appearing approximately 300 ms after the presentation of rare
or surprising, task-relevant stimuli [5]. To evoke the P300,
subjects are asked to observe a random sequence of two types
of stimuli. One stimulus type (the oddball or target stimulus)
appears only rarely in the sequence, while the other stimulus
type (the normal or nontarget stimulus) appears more often.
Whenever the target stimulus appears, a P300 can be observed
in the EEG. This principle was exploited by Farwell and
Donchin in a BCI system which allowed to spell words by
sequentially selecting symbols from a matrix of symbols [6].
Other examples for ERPs that can be used in BCIs are
steady-state visual evoked potentialss (SSVEPs) and motor-
related potentials (MRPs). Descriptions of systems using such
signals can be found in [7] and [8].
B. Oscillatory Brain Activity
Sinusoid like oscillatory brain activity occurs in many
regions of the brain and changes its characteristics according
to the state of subjects, for example between wake and sleep
or between concentrated work and idling. Oscillatory activity
in the EEG is classified into different frequency bands or
rhythms. Typically observable are the delta (1 - 4 Hz), theta
(4 - 8 Hz), alpha and mu (8 - 13 Hz)
1
, beta (13 - 25 Hz), and
gamma (25 - 40 Hz) rhythms.
Among the above mentioned EEG rhythms, especially the
mu-rhythm is of interest because mu-oscillations are decreased
in amplitude when movements of body parts are imagined or
performed. The changes in the mu-rhythm are localized over
the part of the sensorimotor cortex corresponding to the body
part, and so imagined movements of different body parts can
be discriminated. Since the changes in mu- rhythm occurring
in untrained users are usually not strong enough to be detected
by a classification algorithm, feedback training has to be used.
BCI systems employing imagined movements of hands,
feet, or tongue have been mainly introduced by the research
group of Pfurtscheller in Austria [9]. The group of Wolpaw
in the United States has also worked on such systems, and an
impressive sensorimotor rhythm BCI allowing for fast control
of a 2D cursor has been described in [10].
C. Slow Cortical Potentials
Slow cortical potentials (SCPs) are slow voltage shifts in
the EEG occurring in the frequency range 1-2 Hz. Through
feedback training subjects can learn to voluntarily control their
SCPs. The voluntary production of negative and positive SCPs
has been exploited by Birbaumer et al to show that patients
suffering from ALS can use a BCI to control a spelling device
and to communicate with their environment [11].
D. Neuronal Ensemble Activity
Action potentials are thought to be the basic unit of in-
formation in the brain and enable communication between
different neurons. The number of action potentials per time
(the firing rate) can be used in a BCI to predict the behavior of
a subject. For example the firing rate of ensembles of neurons
in the motor and premotor-cortices can be used to predict hand
positions or hand velocities. Neuronal ensemble activity can
thus be employed as neurophysiological signal in BCIs, in
particular in BCIs using microelectrode arrays [12].
1
The term mu-rhythm is used for oscillatory activity with a frequency of
about 10 Hz, localized over the sensorimotor cortex. The term alpha-rhythm
is more general and can be used for any activity in the frequency range 8 -
13 Hz.
III. SIGNAL PROCESSING AND MACHINE LEARNING
In the previous section we have discussed paradigms that
let users control their brain activity and the neurophysiologic
signals corresponding to the respective paradigms. To allow
actual control of a BCI, the neurophysiologic signals have
to be mapped to values that allow to discriminate different
classes of signals, i.e.the neurophysiologic signals have to be
classified. In BCIs, machine learning algorithms are applied
to learn from a training dataset how to classify the signals
of a specific user. As is well known, most machine learning
algorithms can be divided into two modules: signal process-
ing (also known as feature extraction) and classification. In
the following subsections we first review signal processing
methods that are typically used in BCIs and then give a short
introduction to classification methods for BCIs.
A. Time Domain Features
Time domain features are related to changes in the ampli-
tude of neurophysiologic signals, occurring time-locked to the
presentation of stimuli or time-locked to actions of the user of
a BCI. Good examples for signals that can be characterized
with the help of time domain features are the P300 and SCPs.
A strategy that is often used to separate these signals from
background activity and noise is lowpass or bandpass filtering,
optionally followed by downsampling.
An alternative to filtering is to use the wavelet transform of
the signals. Systems based on the discrete wavelet transform
(DWT), as well as systems based on the continuous wavelet
transform (CWT) have been described in the literature.
Besides the use for the EEG signals P300, SCP, and MRP,
time domain features are also used in BCI systems based on
neuronal ensemble activity. A feature that is often used in such
systems is the number of spikes occurring in a certain time
interval. Many different techniques for counting spikes and for
sorting spikes recorded with the same electrode from several
neurons exist. These techniques will however not be further
discussed here.
B. Frequency Domain Features
Frequency domain features are related to changes in oscilla-
tory activity. Since the phase of oscillatory activity is usually
not time-locked to the presentation of stimuli or to actions of
the user, time domain feature extraction techniques cannot be
used. Instead, feature extraction techniques that are invariant
to the exact temporal evolution of signals have to be used.
The most commonly used frequency domain features are
related to changes in the amplitude of oscillatory activity. For
example in systems based on motor imagery, the bandpower
in the mu and beta frequency bands at electrodes located over
the sensorimotor cortex can be used as a feature. To estimate
band power, different methods have been used. These include
Welch’s method [7], adaptive autoregressive models [13], and
Morlet wavelets [14].
A second type of frequency domain features is related to the
synchronization between signals from different brain regions.
Synchronization of signals from different brain regions might
indicate that these regions communicate. This permits to
discriminate cognitive tasks involving communication between
different brain regions [15].
C. Spatial Domain Features
The feature extraction techniques described so far all work
with univariate time series, i.e. data from only one electrode
is used (an exception are synchronization features, extracted
from bivariate time series). In many systems however, data
from more than one electrode is available. Hence, the features
extracted from several electrodes have to be combined in an
efficient way. Finding efficient combinations of features from
more than one electrode is the goal of spatial feature extraction
methods.
The probably simplest way for performing spatial feature
extraction is to use only electrodes that carry useful informa-
tion for discrimination of a given set of cognitive tasks. The
reasoning behind such an approach is that changes in band
power, P300 peaks, or other features do not occur uniformly at
all electrodes but are usually stronger at electrodes over brain
regions implied in the respective cognitive task. Electrodes
can be selected manually or by using an algorithm that
automatically selects an optimal electrode subset.
A spatial feature extraction method that can be used in
addition to electrode selection, consists in applying spatial
filtering algorithms before further processing takes place.
Spatial filtering corresponds to building linear combinations
of the signals measured at several electrodes. Denoting by
s(t) ∈ R
E
the signal from E electrodes at time t, spatial
filtering can be expressed as follows:
ˆs(t)=Cs(t). (1)
Here the F ×E matrix C contains the coefficients for F spatial
filters and the vector
ˆ
s(t) ∈ R
F
contains the spatially filtered
signals at time t.
To determine the filter coefficients different methods can
be used. For example for motor imagery based BCIs, it
has been shown that spatial filtering with a Laplacian filter
can increase performance [16]. In other methods for spatial
feature extraction, filter coefficients are computed from a set
of training data. An algorithm which is very popular in the
area of motor imagery based BCI systems is the common
spatial patterns (CSP) algorithm [17]. Another method for
computing the coefficients of spatial filters from training data
is independent component analysis (ICA). The assumption
underlying the application of ICA to EEG signals is that the
signals measured on the scalp are a linear and instantaneous
mixture of signals from independent sources in the cortex,
deeper brain structures, and noise [18]. ICA has been mainly
used in P300-based BCIs as a feature extraction method (see
for example [19]).
D. Machine Learning
After feature extraction with one of the methods mentioned
above (or with a combination of methods), supervised machine
learning algorithms are applied to learn how to classify the
signals of a specific user.
A simple but efficient method for supervised machine
learning, appropriate for use in BCIs, is Fisher’s discriminant
analysis (FDA) (see for example [20]). The main advantages
of FDA are that it is a computationally and conceptually
simple method and that very good classification accuracy can
be achieved. A possible drawback of FDA is that a squared-
error loss function is used which makes the method vulnerable
to outliers in the training data. Furthermore, a precondition for
using FDA is that the number of training examples is higher
than the number of dimensions of the training data. In BCI
applications it can happen that this precondition is not fulfilled.
A remedy to the aforementioned problems is to use regularized
FDA [21]. A Bayesian, regularized version of FDA is briefly
described in Section V-C of this paper.
Another algorithm that is relatively often used in BCIs is the
support vector machine (SVM) [20]. The main advantages of
the SVM are that it allows to achieve very good classification
accuracy and that nonlinear classification functions can be eas-
ily implemented by using kernels. A drawback is however, that
training SVMs is computationally complex because regulariza-
tion constants and kernel parameters are typically estimated
with a cross-validation procedure. A second issue is that the
loss function used in the SVM is designed for problems in
which only binary yes/no outputs are needed. The problem
with binary yes/no outputs is that no information is given about
the confidence the system has in those outputs. Confidence
information, or alternatively probabilistic classifier outputs, are
however important to build real-world BCI applications.
Besides FDA and SVM many other machine learning al-
gorithms have been tested in the context of BCI systems. An
overview of these algorithms can be found in [22].
IV. A
PPLICATIONS
In this section we give examples of applications that can be
controlled with a BCI.
A. Spelling Devices
Spelling devices allow severely disabled users to communi-
cate with their environment by sequentially selecting symbols
from the alphabet. One of the first spelling devices mentioned
in the BCI literature is the P300 speller [6]. Another system,
tested with users suffering from ALS and based on SCPs was
described by Birbaumer [11].
B. Environment Control
Environment control systems allow to control electrical ap-
pliances with a BCI. A proof-of-concept environment control
system based on SSVEPs is described in [23]. The control of
a virtual apartment with a BCI using the P300 is described in
[24].
C. Wheelchair Control
A BCI can potentially be used to steer a wheelchair.
Because steering a wheelchair is a complex task and because
wheelchair control has to be extremely reliable, the possible
movements of the wheelchair are strongly constrained in cur-
rent prototype systems. For example in the system presented
in [25], the wheelchair is constrained to move along paths
predefined in software, joining registered locations, and a
P300-based interface is used to select the desired location.
D. Neuromotor Prostheses
The idea underlying research on neuromotor prostheses is
to use a BCI for controlling movement of limbs and to restore
motor function in tetraplegics or amputees. Different types
of neuromotor prostheses can be envisioned depending on the
information transfer rate a BCI provides. If neuronal ensemble
activity is used as control signal, high information transfer
rates are achieved and 3D robotic arms can be controlled [26].
If an EEG based BCI is used, only simple control tasks can
be accomplished. For example in the system described by [27]
sensorimotor rhythms were used to control functional electric
stimulation of hand muscles and so to restore grasp function
in a tetraplegic patient.
E. Gaming and Virtual Reality
Besides the applications targeted towards disabled subjects,
prototypes of gaming and virtual reality applications have been
described in the literature. Examples for such applications are
the control of a spaceship with oscillatory brain activity [28]
and the control of an animated character in an immersive 3D
gaming environment with SSVEPs [7].
V. A
N EFFICIENT BRAIN-COMPUTER INTERFACE FOR
DISABLED SUBJECTS
After the general review of neurophysiologic signals, signal
processing and machine learning methods, and BCI applica-
tions, we now turn our attention to a more detailed description
of a state-of-the-art BCI system. More specifically, a BCI
system using the P300, developed in the Multimedia Signal
Processing Group at the EPFL, is described. The interested
reader can find more details about this system in [29] and
[30].
A. System Description
In the BCI system developed at the EPFL users are facing a
laptop screen on which six images are displayed (see Fig. 2).
The images show a television, a telephone, a lamp, a door,
a window, and a radio. The images are selected according to
an application scenario in which users can control electrical
appliances via a BCI system. The application scenario serves
however only as an example and is not pursued in further
detail.
The images are flashed in random sequences, one image at
a time. Each flash of an image lasts for 100 ms and during
the following 300 ms none of the images is flashed, i.e. the
interstimulus interval (ISI) is 400 ms. The EEG is recorded
at 2048 Hz sampling rate from thirty-two electrodes placed
at the standard positions of the 10-20 international system.
A Biosemi Active Two amplifier is used for amplification
and analog to digital conversion of the EEG signals. Signal
processing and machine learning algorithms are implemented
with MATLAB. The stimulus display and the online access
to the EEG signals are implemented as dynamic link libraries
(DLLs) in C. The DLLs are accessed from MATLAB via a
MEX interface.
B. Materials and Methods
The system is tested with five disabled and four able-bodied
subjects. The disabled subjects suffer from different diseases
such as multiple sclerosis or ALS and are all wheelchair-bound
but have varying communication and limb muscle control
abilities (see [29], [30] for more details). The able-bodied
subjects are PhD students recruited from our laboratory. None
of the able-bodied subjects has known neurological deficits.
Each subject completes four recording sessions. The first
two sessions are performed on one day and the last two
sessions on another day. For all subjects the time between
the first and the last session is less than two weeks. Each of
the sessions consists of six runs, one run for each of the six
images. The following protocol is used in each of the runs.
1) Subjects are asked to count silently how often a pre-
scribed image is flashed (For example: ”Now please
count how often the image with the television is
flashed”).
2) The six images are displayed on the screen and a
warning tone is issued.
3) Four seconds after the warning tone, a random sequence
of flashes is started and the EEG is recorded. The
sequence of flashes is block-randomized, this means
that after six flashes each image is flashed once, after
twelve flashes each image is flashed twice, etc.. The
number of blocks is chosen randomly between 20 and
25. On average 22.5 blocks of six flashes are displayed
in one run, i.e. one run consists on average of 22.5 target
(P300) trials and 22.5 · 5 = 112.5 nontarget (non P300)
trials.
4) In the second, third, and fourth session the target image
Fig. 2. The display used for evoking the P300. Images were flashed, one at
a time, by changing the overall brightness of images.
is inferred from the EEG with a simple classifier
2
. At the
end of each run the image inferred by the classification
algorithm is flashed five times to give feedback to the
user.
5) After each run subjects are asked what their counting
result is. This is done in order to monitor performance
of the subjects.
The duration of one run is approximately one minute and
the duration of one session including setup of electrodes and
short breaks between runs is approximately 30 minutes. One
session comprises on average 810 trials, and the whole data
for one subject consists on average of 3240 trials.
C. Signal Processing and Machine Learning
To extract features from the raw EEG signals, the following
operations are applied:
1) Referencing
The average signal from the two mastoid electrodes is
used for referencing.
2) Filtering
A 6th order forward-backward Butterworth bandpass
filter is used to filter the data. Cutoff frequencies are set
to 1.0 Hz and 12.0 Hz. The MATLAB function butter is
used to compute the filter coefficients and the function
filtfilt is used for filtering.
3) Downsampling
The EEG is downsampled from 2048 Hz to 32 Hz by
selecting each 64th sample from the bandpass-filtered
data.
4) Single Trial Extraction
Single trials of duration 1000 ms are extracted from
the data. Single trials start at stimulus onset, i.e. at the
beginning of the intensification of an image, and end
1000 ms after stimulus onset. Due to the ISI of 400 ms,
the last 600 ms of each trial overlap with the first 600
ms of the following trial.
5) Windsorizing
Eye blinks, eye movement, muscle activity, or subject
movement can cause large amplitude outliers in the
EEG. To reduce the effects of such outliers, the data
from each electrode are windsorized. For the samples
from each electrode the 10th percentile and the 90th
percentile are computed. Amplitude values lying below
the 10th percentile or above the 90th percentile are then
replaced by the 10th percentile or the 90th percentile,
respectively.
6) Scaling
The samples from each electrode are scaled to the
interval [−1, 1].
7) Electrode Selection
For the results presented in this paper data from a set of
eight electrodes is used. The electrode set consists of the
2
The classifier is trained from the data recorded in the first session. The
algorithm described in [31] is used for preprocessing and the algorithm
described in [32] is used for classification.