This is the Accepted Manuscript version of an article accepted for publication in Journal of Neural Engineering. IOP Publishing
Ltd is not responsible for any errors or omissions in this version of the manuscript or any version derived from it. The Version of
Record is available online at: Krol, L. R., Haselager, P., & Zander, T. O. (2020). Cognitive and affective probing: a tutorial and
review of active learning for neuroadaptive technology. Journal of Neural Engineering, 17 (1), 012001. doi:
10.1088/1741-2552/ab5bb5
Cognitive and Affective Probing: A Tutorial and Review of
Active Learning for Neuroadaptive Technology
Laurens R. Krol
1,*
, Pim Haselager
2
, and Thorsten O. Zander
3
1
Biological Psychology and Neuroergonomics, Technische Universit¨at Berlin,
Berlin, Germany
2
Donders Institute for Brain, Cognition and Behaviour, Radboud University,
Nijmegen, The Netherlands
3
Zander Laboratories B.V., Amsterdam, The Netherlands
*
Correspondence: lrkrol@gmail.com
Abstract
Objective: The interpretation of neurophysiological measurements has a decades-
long history, culminating in current real-time brain-computer interfacing (BCI) ap-
plications for both patient and healthy populations. Over the course of this history,
one focus has been on the investigation of cortical responses to specific stimuli. Such
responses can be informative with respect to the human user’s mental state at the
time of presentation. An ability to decode neurophysiological responses to stim-
uli in real time becomes particularly powerful when combined with a simultaneous
ability to autonomously produce such stimuli. This allows a computer to gather
stimulus-response samples and iteratively produce new stimuli based on the infor-
mation gathered from previous samples, thus acquiring more, and more specific,
information. This information can even be obtained without the explicit, voluntary
involvement of the user.
Approach: We define cognitive and affective probing, referring to an applica-
tion of active learning where repeated sampling is done by eliciting implicit brain
responses. In this tutorial, we provide a definition of this method that unifies differ-
ent past and current implementations based on common aspects. We then discuss
a number of aspects that differentiate various possible implementations of cognitive
probing.
Main results: We argue that a key element is the user model, which serves as
both information storage and basis for subsequent probes. Cognitive probing can
be used to continuously and autonomously update this model, refining the probes,
and obtaining increasingly detailed or accurate information from the resulting brain
activity. In contrast to a number of potential advantages of the method, cognitive
probing may also pose a threat to informed consent, our privacy of thought, and
our ability to assign responsibility to actions mediated by the system.
Significance: This tutorial provides guidelines to both implement, and critically
discuss potential ethical implications of, novel cognitive probing applications and
research endeavours.
1 Introduction
One essential skill required of disc jockeys (DJs) is the ability to “read the crowd”. In
order to maximise the audience’s enjoyment, a DJ may play different songs and gauge
1
This is the Accepted Manuscript version of an article accepted for publication in Journal of Neural Engineering. IOP Publishing
Ltd is not responsible for any errors or omissions in this version of the manuscript or any version derived from it. The Version of
Record is available online at: Krol, L. R., Haselager, P., & Zander, T. O. (2020). Cognitive and affective probing: a tutorial and
review of active learning for neuroadaptive technology. Journal of Neural Engineering, 17 (1), 012001. doi:
10.1088/1741-2552/ab5bb5
the reaction they evoke (Gates, Subramanian, & Gutwin, 2006). Depending on this
reaction, the DJ decides whether or not to end the current song early, and what to play
next. This is an iterative process that continues to optimise with each played song, the
goal being maximum audience participation, or indeed, maximum audience happiness.
The feedback from the audience is essential input in this process. However, the au-
dience members themselves are not necessarily aware of providing any—their responses
are mostly automatic, even reflexive. Hardly would one deliberate whether or not the
current chords are sufficiently engaging; rather, the rhythm and mood of the music will
simply induce the audience members to express themselves in a certain way, and it is
this expression that the DJ will read and use as input.
In this tutorial, we will discuss a situation analogous to the one above, in the context
of human-computer interaction (HCI): An adaptive process where a computer presents
something that automatically evokes a response from the user, which the computer can
then interpret and use for further processing. We will discuss the methods involved,
how they relate to past and current applications, and what aspects must be considered
going forward with this technology.
Let us first discuss what an “automatic” response may be in the context of HCI.
In traditional HCI, all input that we give to the computer is based on goal-oriented,
explicit, voluntary actions. By explicit communication, we mean any intentional action
performed for the purpose of communicating specific content. For example, to commu-
nicate information to a computer, we must actively type, say, click, move, swipe, or
select things in a corresponding fashion, or nothing will happen.
There are technologies that give a computer access to more information about the
user than what is explicitly communicated. In virtual reality, moving our arms and legs
provides explicit positional information to the computer, but, going beyond that, full-
body motion tracking can also be used to analyse these movements, or the resulting pos-
tures, for additional information that was not meant to be communicated (Kleinsmith,
Bianchi-Berthouze, & Steed, 2011). Your movements may reveal that you are happy or
sad, or perhaps drunk. Similarly, for example, facial recognition can be used to assess
emotions (Janssen et al., 2013), and passive brain-computer interfaces can analyse our
brain activity in order to supply implicit input to a computer (Zander & Kothe, 2011;
Krol, Andreessen, & Zander, 2018). By implicit communication, we mean that informa-
tion is acquired from output that was not intended to communicate that information.
Here, we focus on methods that target our brains—the primary seat of cognition and
conscious experience—as they are uniquely positioned to infer information about us in
such a way.
Real-time detection of mental states has been demonstrated as early as the 1970s
(Vidal, 1977), and since then, further research has shown that BCI methodology can
be used to detect a range of different patterns of brain activity, including some that
reflect specific cognitive and affective processes (e.g., Donchin, 1981; Hillyard & Anllo-
Vento, 1998; Yeung & Sanfey, 2004). These mental processes occur naturally during
our everyday lives. So-called passive BCI systems (Zander & Kothe, 2011; Krol et al.,
2018) target such naturally-occurring mental states in order to provide implicit input to
a computer (Schmidt, 2000; Zander, Br¨onstrup, Lorenz, & Krol, 2014). In other words,
passive BCI systems rely on brain activity that was not intended by the user to provide
information to those systems, but is nonetheless used as such.
It should be noted, however, that our current ability to discern different mental
states, and thus our ability to infer specific information from brain activity, is limited.
While research into “reading” mental states in novel ways continues to move forward
2
This is the Accepted Manuscript version of an article accepted for publication in Journal of Neural Engineering. IOP Publishing
Ltd is not responsible for any errors or omissions in this version of the manuscript or any version derived from it. The Version of
Record is available online at: Krol, L. R., Haselager, P., & Zander, T. O. (2020). Cognitive and affective probing: a tutorial and
review of active learning for neuroadaptive technology. Journal of Neural Engineering, 17 (1), 012001. doi:
10.1088/1741-2552/ab5bb5
(e.g., Huth, de Heer, Griffiths, Theunissen, & Gallant, 2016; Haynes, 2011; Senden,
Emmerling, van Hoof, Frost, & Goebel, 2019), it is not currently possible to infer the
propositional content of thought (Haselager & Mecacci, 2018). Thus, in many cases, ad-
ditional information about the current situation is required in order to draw meaningful
inferences from measured brain activity (Makeig, Gramann, Jung, Sejnowski, & Poizner,
2009; Zander & Jatzev, 2012). For example, when high workload is detected in a multi-
tasking environment, additional information is required to make an optimal decision
on which task to automate (M¨uhl, Jeunet, & Lotte, 2014; Krol, Freytag, Fleck, Gra-
mann, & Zander, 2016). Similarly, when error processing is detected, it is important to
know where that error originated (Falkenstein, Hoormann, Christ, & Hohnsbein, 2000;
Mousavi & de Sa, 2019; Wirth, Dockree, Harty, Lacey, & Arvaneh, 2019). Determining
the best way for a computer to sense, filter, and represent relevant situational aspects
remains an open field of research with many different approaches (Perera, Zaslavsky,
Christen, & Georgakopoulos, 2014).
In our earlier analogy, if an observer to the DJ’s performance sees the crowd’s reaction
but cannot hear and does not know what music is currently playing, they will be none
the wiser, until they obtain this extra piece of information. The DJ, on the other hand,
doesn’t only know what music is playing—she is the one who selected the music in the
first place!
Similar to a DJ, in HCI contexts where a computer has control over relevant situa-
tional aspects as well as access to mental state measurements, it can also actively induce
an even in order to assess the user’s brain response to it. This allows it to probe the hu-
man user. Having a computer induce an event itself circumvents the need for elaborate
post-hoc context sensing. Rather than waiting for events to occur naturally in order
to learn the relation between events and mental states, the computer can purposefully
generate specific events and gauge the user’s implicit response. Doing this in an iterative
fashion allows the computer to adaptively pursue a particular goal. As we will see, these
events can also be “hidden”, embedded in the ongoing interaction itself. Furthermore,
considering that humans are generally unable to stop their brains from automatically
processing perceived stimuli, the computer is effectively posing a question directly to
the user’s brain, potentially bypassing the user’s explicit faculties.
In essence, this cognitive probing can be seen as a form of active learning (Settles,
2009). Active learning is a concept in machine learning where a learner, rather than
simply accepting samples as they come, has control over which samples it learns from.
Cognitive probing applies this concept to a computer learning from implicit responses
to technological state changes, with the added notion that the learner in this case is not
merely responsible for its own learning, but also for the effect its sampling has on the
human. Inducing mental states is not a neutral act.
Learning itself can be the sole goal of the computer. However, additional goals
may be pursued as well: much like the DJ’s goal of maximum audience happiness, the
computer, too, may attempt to find stimuli that maximise certain psychological states.
As such, cognitive probing can be a powerful tool to realise closed-loop neuroadaptive
technology. Neuroadaptive technology is technology that adapts itself based on implicit
input of neurophysiological origin (Zander & Krol et al., 2016). The amount of in-
put obtained by such technology, and the efficiency with which it is obtained, can be
greatly increased using cognitive probing. In particular, the technology can iteratively
adapt itself to the user based on implicit information gathered from probe-induced brain
activity.
The use of passive BCIs for cognitive probing thus provides a potentially powerful
3
This is the Accepted Manuscript version of an article accepted for publication in Journal of Neural Engineering. IOP Publishing
Ltd is not responsible for any errors or omissions in this version of the manuscript or any version derived from it. The Version of
Record is available online at: Krol, L. R., Haselager, P., & Zander, T. O. (2020). Cognitive and affective probing: a tutorial and
review of active learning for neuroadaptive technology. Journal of Neural Engineering, 17 (1), 012001. doi:
10.1088/1741-2552/ab5bb5
method for a computer to unobtrusively obtain information about its users. At the same
time, it also represents a potentially dangerous and psychologically invasive method
concerning mental privacy, and hence its application must be handled with due care
and transparency (Mecacci & Haselager, 2019b).
In this tutorial, we first introduce and explain a general, broad definition of this
method. We provide examples from the literature, pointing out that cognitive prob-
ing, as defined here, has already been used for at least three decades in different ways
and different fields. With recent advances in neurophysiological sensing (Mullen et al.,
2015; Zander et al., 2017) and neuroadaptive technology (Zander & Krol et al., 2016;
Lorenz, Hampshire, & Leech, 2017), however, we believe it is prudent to now unify these
methods previously considered disparate in a single framework, and to present a careful
consideration of the different aspects involved. Therefore, a further section discusses a
list of aspects, characteristics, and other considerations that are relevant to the use of
cognitive probing in different applications, and to society at large. We conclude with a
speculative vision of the future, highlighting possible research endeavours.
2 Cognitive Probing
At the basis of our proposed definition lies a single probe, where:
A cognitive (or affective) probe is a technological state change that is
initiated or manipulated by the same technology that uses it to learn from
the user’s implicit, situational, cognitive or affective brain response to it.
Cognitive probing or affective probing, then, is the repeated use of probes in order
to generate a model in which the learned information is registered.
1
We use cognitive probing as the general term including both cognitive and affective
probing; the term affective probing can be used in cases focusing exclusively on affect
(Zander & Jatzev, 2009). Also note that the base definition refers to a single probe,
i.e. a single state change eliciting a single response. Technically, meaningful learning
can be done using a single probe: given sufficiently accurate tools, a single probe may
be sufficient to provide adequate information. However, the main potential, and main
focus, of the method lies in the adaptive, sequential use of multiple probes.
We will now discuss the definition’s constituent elements in more detail.
Technological state change. A technological state change is any action undertaken
by a piece of technology. We use the term technology to collectively refer to any and all
(connected) technological elements in any potential situation. With a technology’s state
being its specific current configuration, a technological state change is thus any change
in configuration that the technology undergoes.
In neurophysiological experiments, the presentation of cues, stimuli, or feedback
would be an example of a state change. In the example of an automated, neuroadaptive
DJ, it could be a song that is started.
Initiated or manipulated. Most technological state changes that a user will notice
are user-generated, i.e. commanded by the user themselves, with the technology reacting
1
The definitions given here for cognitive probe and cognitive probing supersede the earlier working
definitions presented in Krol and Zander (2018).
4
