Digitizing the chemical senses: possibilities & pitfalls
Article (Accepted Version)
http://sro.sussex.ac.uk
Spence, Charles, Obrist, Marianna, Velasco, Carlos and Ranasinghe, Nimesha (2017) Digitizing
the chemical senses: possibilities & pitfalls. International Journal of Human-Computer Studies,
107. pp. 62-74. ISSN 1071-5819
This version is available from Sussex Research Online: http://sro.sussex.ac.uk/id/eprint/69111/
This document is made available in accordance with publisher policies and may differ from the
published version or from the version of record. If you wish to cite this item you are advised to
consult the publisher’s version. Please see the URL above for details on accessing the published
version.
Copyright and reuse:
Sussex Research Online is a digital repository of the research output of the University.
Copyright and all moral rights to the version of the paper presented here belong to the individual
author(s) and/or other copyright owners. To the extent reasonable and practicable, the material
made available in SRO has been checked for eligibility before being made available.
Copies of full text items generally can be reproduced, displayed or performed and given to third
parties in any format or medium for personal research or study, educational, or not-for-profit
purposes without prior permission or charge, provided that the authors, title and full bibliographic
details are credited, a hyperlink and/or URL is given for the original metadata page and the
content is not changed in any way.
DIGITIZING THE CHEMICAL SENSES 1
RUNNING HEAD: DIGITIZING THE CHEMICAL SENSES
Digitizing the chemical senses:
Possibilities & pitfalls
Charles Spence (University of Oxford),
Marianna Obrist (University of Sussex),
Carlos Velasco (BI Norwegian Business School),
& Nimesha Ranasinghe (National University of Singapore)
WORD COUNT: 16,150 WORDS
RESUBMITTED TO: INTERNATIONAL JOURNAL OF HUMAN-COMPUTER STUDIES
DATE: JUNE, 2017
CORRESPONDENCE TO: Prof. Charles Spence, Department of Experimental Psychology,
University of Oxford, Oxford, OX1 3UD, UK. E-mail: charles.spence@psy.ox.ac.uk
DIGITIZING THE CHEMICAL SENSES 2
ABSTRACT
Many people are understandably excited by the suggestion that the chemical senses can be
digitized; be it to deliver ambient fragrances (e.g., in virtual reality or health-related
applications), or else to transmit flavour experiences via the internet. However, to date,
progress in this area has been surprisingly slow. Furthermore, the majority of the attempts at
successful commercialization have failed, often in the face of consumer ambivalence over the
perceived benefits/utility. In this review, with the focus squarely on the domain of Human-
Computer Interaction (HCI), we summarize the state-of-the-art in the area. We highlight the
key possibilities and pitfalls as far as stimulating the so-called ‘lower’ senses of taste, smell,
and the trigeminal system are concerned. Ultimately, we suggest that mixed reality solutions
are currently the most plausible as far as delivering (or rather modulating) flavour
experiences digitally is concerned. The key problems with digital fragrance delivery are
related to attention and attribution. People often fail to detect fragrances when they are
concentrating on something else; And even when they detect that their chemical senses have
been stimulated, there is always a danger that they attribute their experience (e.g., pleasure) to
one of the other senses – this is what we call ‘the fundamental attribution error’. We conclude
with an outlook on digitizing the chemical senses and summarize a set of open-ended
questions that the HCI community has to address in future explorations of smell and taste as
interaction modalities.
KEYWORDS: CHEMICAL SENSES; TASTE; SMELL; TRIGEMINAL; CROSSMODAL;
VIRTUAL REALITY; AUGMENTED REALITY; MIXED REALITY.
DIGITIZING THE CHEMICAL SENSES 3
Introduction
Both the popular press and the general public are fascinated by the possibilities associated
with the digitization of the chemical senses (e.g., Berenstein, 2015; Cuthbertson, 2015; Lant
& Norman, 2017; Marks, 2013; Obrist, Tuch, & Hornbæk, 2014; Platt, 1999). And indeed,
much research has been conducted in this area in recent decades (as documented by the many
papers and sessions at the technology conferences in HCI such as ACM CHI, UIST, and TEI,
not to mention, by the various papers referenced in this review). That said, there are still a
number of key questions with regard to the digitization of the chemical senses that will need
to be addressed before any real progress can be made in delivering plausible (i.e.,
commercially viable and appealing) solutions to market. These include: What do you want to
digitize? Why do you want to digitize it? How do you plan to digitize it? What are the
limitations, both technical and psychological, to digital transmission/delivery that are
relevant to the chemical senses? It is only by addressing such questions that the various
pitfalls that have been highlighted by a number of the high-profile failures in this area in
recent years can be avoided (e.g., Dusi, 2014; Twilley, 2016; Velasco, Obrist, Petit,
Karunanayaka, Cheok, & Spence, 2016).
Introducing the chemical senses
At the outset, when thinking about the digitization of the chemical senses, it is important to
note that there are at least three senses that may be the target of any digital intervention: 1)
Stimulation of the sense of taste (gustation); 2) Stimulation of the sense of smell (olfaction
via either the orthonasal or retronasal route; e.g., Rozin, 1982; Small, Gerber, Mak, &
Hummel, 2005); and 3) trigeminal stimulation (responsible for detecting sensations such as
heat and cold along with various food textures that are related to biting and chewing actions;
e.g., Burdach, Kroeze, & Koster, 1984; Dodd & Kelly, 1991; Lundström, Boesveldt, &
Albrecht, 2011; Spence & Piqueras-Fiszman, 2016; Viana, 2011).
The delivery of ambient scent is the simplest application of digitizing the chemical senses,
since it requires only orthonasal olfactory stimulation (e.g., as when we inhale/sniff). Such
scents might or might not be food (i.e., flavour) related. To date, digitally-controlled scent
delivery
1
have been used to augment the immersion in audio-visual entertainment/training
applications (Cole, 2016; see Ischer, Baron, Mermoud, Cayeux, Porcherot, Sander, &
Delplanque, 2014, for a review). More generally, ambient scents have been used to trigger
specific moods, emotions (Herz, 2002; Leenders, Smidts, & El Haji, in press; Moss, Cook,
Wesnes, & Duckett, 2003; Rétiveau, Chambers, & Milliken, 2004), nostalgia/memories (Chu
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1
When considering the digitization of smell, it would seem natural to consider digitally-controlled chemical
delivery systems, electric (or perhaps thermal) stimulation, or a combination of both. Chemicals are naturally
processed by our olfactory system, nevertheless, researchers have also explored direct electric stimulation of the
olfactory receptors as means to evoke odour sensations (Hariri, Mustafa, Karunanayaka, & Cheok, 2016). It is
important to recognize, though, that such stimulation does not give rise to the perception of odours as does
following chemical stimulation (Weiss et al., 2016). In that sense, as for to date, it seems more plausible to
digitizing the sense of smell via digitally-controlled chemicals.
DIGITIZING THE CHEMICAL SENSES 4
& Downes, 2000, 2002; Doop, Mohr, Folley, Brewer, & Park, 2006; Tortell, Luigi, Dozois,
Bouchard, Morie, & Ilan, 2007), induce hunger, and even bias our everyday behaviours
(Holland, Hendriks, & Aarts, 2005).
By contrast, stimulation of the sense of taste, retronasal olfaction,
2
and possibly also the
trigeminal sense are needed in order to deliver an authentic-tasting flavour experiences (e.g.,
Bult, de Wijk, & Hummel, 2007; Piqueras-Fiszman & Spence, 2016). Just think, for example,
about simulating the minty sensation associated with compounds such as 1-methol (the
principal flavour in mint). All three of these sensory systems are needed if one is to deliver
the characteristic minty aroma, the slightly bitter taste, and the cooling mouth-feel (involving
the tactile thermal nociceptors) associated with the experience of this particular stimulus
(Nagata, Dalton, Doolittle, & Breslin, 2005). Of course, it is not enough simply to stimulate
these senses; The relative intensity of these digital stimuli also needs to be right, as does the
time-course of increasing and decreasing sensation (see Obrist, Comber, Subramanian,
Piqueras-Fiszman, Velasco, & Spence, 2014; Stuckey, 2012), if one wants to simulate a
genuinely-compelling (i.e., authentic) minty sensation.
Taste (strictly-speaking, gustation) and flavour (the latter referring to the combined input of
gustatory, olfactory, and possibly also trigeminal stimulation) are undoubtedly complex
concepts to try and disentangle, both at the theoretical and at the empirical levels (see Spence,
Smith, & Auvray, 2015, for a review). Matters are made more confusing by the existence of
phenomenon such as oral referral (of odours to the oral cavity; see Spence, 2016a, for a
review), and the fact that different terms are sometimes used in different languages to refer to
these two percepts (e.g., Rozin, 1982; Spence, 2017a). Here it is perhaps helpful to bear in
mind that stimulation of the taste-buds on the human tongue may only give rise to the
sensation of sweet, bitter, salty, sour, and umami.
3
Everything else that we enjoy while tasting
– the meaty, the fruity, the floral, the herbaceous, and the roasted etc. – are all delivered by
the sense of smell instead.
4
That is, by volatile molecules hitting the olfactory receptors
embedded in the nasal mucosa. It is one of the tricks of the mind that so much of this
information, transduced by the olfactory receptors in the nose is referred to the mouth, giving
us all the illusion that we are tasting (this is what it is referred to as ‘oral referral’). So, when
talking about the digitization of the chemical senses, one needs to keep taste distinct from
tasting (the latter normally used to refer to the flavour perceived; see Spence et al., 2015). It
is worth bearing in mind that it has widely been estimated that 75-95% of what we think we
taste really reflects information delivered by the sense of smell (see Spence, 2015a, for a
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2
Retronasal olfaction is based on the volatile-rich air that is pulsed out from the back of the nose whenever we
swallow (e.g., Bojanowski & Hummel, 2012).
3
That said, a growing number of researchers now believe that oleogustus, or fatty acid, should be considered as
the sixth taste (e.g., Keast & Costanzo, 2015; Running, Craig, & Mattes, 2015). Then there is kokumi, not to
mention the recently-discovered taste for glucose oligomers (Lapis, Penner, & Lim, 2016). However, while we
may well be able to discriminate these stimuli in taste tests, it is not so clear that they are all necessarily
associated with a clearly identifiable taste percept.
4
It is currently unclear whether the metallic sensation one sometimes gets is a taste, a retronasal aroma, or a
flavour (see Skinner, Lim, Tarrega, Ford, Linforth, & Hort, in press; Spence et al., 2015).
![Figure 1. Sketch (on the left) and picture (on the right) of the Sensorama Simulator patented by M. L. Heilig (1962). This invention is widely credited as being the first simulator designed to stimulate multiple senses. [Figure reproduced from Heilig (1962, Figure 5), Patent 3,050,870.]](/figures/figure-1-sketch-on-the-left-and-picture-on-the-right-of-the-2do0u4vr.webp)
![Figure 4. Digital seasoning from Huisman et al. (2016). Colour, shape, and sound are all used to prime notions of sourness, and by so doing modify people’s perception of the sourness of the yoghurt that they are tasting. [Figure reprinted from Huisman et al. (2016) with permission.]](/figures/figure-4-digital-seasoning-from-huisman-et-al-2016-colour-rketif3u.webp)