Of epistemic tools: musical instruments as cognitive
extensions
Article (Published Version)
http://sro.sussex.ac.uk
Magnusson, Thor (2009) Of epistemic tools: musical instruments as cognitive extensions.
Organised Sound, 14 (2). pp. 168-176. ISSN 1355-7718
This version is available from Sussex Research Online: http://sro.sussex.ac.uk/id/eprint/46873/
This document is made available in accordance with publisher policies and may differ from the
published version or from the version of record. If you wish to cite this item you are advised to
consult the publisher’s version. Please see the URL above for details on accessing the published
version.
Copyright and reuse:
Sussex Research Online is a digital repository of the research output of the University.
Copyright and all moral rights to the version of the paper presented here belong to the individual
author(s) and/or other copyright owners. To the extent reasonable and practicable, the material
made available in SRO has been checked for eligibility before being made available.
Copies of full text items generally can be reproduced, displayed or performed and given to third
parties in any format or medium for personal research or study, educational, or not-for-profit
purposes without prior permission or charge, provided that the authors, title and full bibliographic
details are credited, a hyperlink and/or URL is given for the original metadata page and the
content is not changed in any way.
http://journals.cambridge.org Downloaded: 01 Jul 2009 IP address: 139.184.30.131
Of Epistemic Tools: musical instruments
as cognitive extensions
THOR MAGNUSSON
ixi audio, Music Informatics Research Lab, Department of Informatics, University of Sussex, BN1 9RH Brighton
E-mail: thor@ixi-audio.net
This paper explores the differences in the design and
performance of acoustic and new digital musical instruments,
arguing that with the latter there is an increased
encapsulation of musical theory. The point of departure is the
phenomenology of musical instruments, which leads to the
exploration of designed artefacts as extensions of human
cognition – as scaffolding onto which we delegate parts of our
cognitive processes. The paper succinctly emphasises the
pronounced epistemic dimension of digital instruments when
compared to acoustic instruments. Through the analysis of
material epistemologies it is possible to describe the digital
instrument as an epistemic tool: a designed tool with such a
high degree of symbolic pertinence that it becomes a system
of knowledge and thinking in its own terms. In conclusion, the
paper rounds up the phenomenological and epistemological
arguments, and points at issues in the design of digital
musical instruments that are germane due to their strong
aesthetic implications for musical culture.
1. INTRODUCTION
The philosopher Don Ihde is well known for his
analysis of how we establish a relationship with the
world through tool use (Ihde 1979, 1990). Ihde reports
on various phenomenological modalities in our rela-
tionship with tools, two of which are relevant to this
article. For Ihde, the acoustic musical instrument is an
illustrative example of a technology that enables an
embodiment relationship to the world. The instrument
becomes an extension of the body, where trained
musicians are able to express themselves through
incorporated knowledge that is primarily non-
conceptual and tacit. The other phenomenological
mode, the hermeneutic relationship, differs in the sense
that here the instrument is not an extension of the
body, but rather a tool external to the body whose
information we have to interpret (thus hermeneutic).
This instrument can be seen as a text, something we
have to read in our use of it. Disregarding the perils of
dualism, and acknowledging that these distinct rela-
tionships can overlap in the same musical instrument,
I would like to propose that many digital instruments
are to be seen primarily as extensions of the mind
rather than the body. This seemingly dichotomous
statement will be explored in section 3 below.
Furthermore, and in relation to the above, I will
argue that while acoustic instruments afford a strong
embodiment relationship with the world (or the
terminus of our activities – the physical energy of
sound), digital instruments increasingly tend to con-
strue us in a hermeneutic relationship with the world.
The tangi ble user interfaces that apparently con stitute
many digital musical instruments are but arbitrary
peripherals of the instruments’ core – that is, a core
that is essentially a symbolic system of computational
design. I contend that the primary body of the digital
instrument is that of symbolic instructions written
for the meta-machine, the computer. As opposed to
the body of the acoustic instrument, the digital
instrument does not resonate; it contains few latent
mysteries, or hidden expressive potential that typi-
cally can be derived from the materiality of acoustic
instruments (Edens 2005). The functionality of the
digital instrument is always explicitly designed and
determined. Indeed, the digital musical instrument –
especially if it makes use of automation or other
mappings that are not one-to-one gesture-to-sound –
is constituted by generic, prescriptive and normative
sets of rules that affect or direct the musician at the
high level of musical language (both formal and
theoretical).
The focus of this paper are novel digital musical
interfaces, in particular those to be found in a
research field best represented by the NIME (New
Interfaces for Musical Expression) conference series.
Although relevant as well, the analysis is not directed
at digital pianos or pure studio simulators like Pro-
Tools. What is of interest are the computational
music systems used to build expressive intelligent
instruments, or composed instruments (Schnell and
Battier 2002), where the distinction often blurs
between instrument and composition on the one
hand, and performance and composition on the
other. Composed instruments typically contain auto-
mation of musical patterns (whether blind or intelli-
gent) that allow the performer to delegate musical
actions to the instrument itself, such as playing
arpeggios, generating rhythms, expressing spatial
dimensions as scales (as opposed to pitches), and so
on. These systems are therefore split systems between
the physical interface and the programmed sound
engine. Typically such engines are programmed in
environments like Pure Data, SuperCollider, ChucK,
Organised Sound 14(2): 168–176 & 2009 Cambridge University Press. Printed in the United Kingdom. doi:10.1017/S1355771809000272
http://journals.cambridge.org Downloaded: 01 Jul 2009 IP address: 139.184.30.131
Max/MSP or Kyma. These tools have such ad vanced
algorithmic capabilities that there is no need to limit
the instrument to direct one-to-one gesture-to-sound
mapping; the instrument can implement various
complex (and even adaptive) mapping structures and
contain various degrees of automation from simple
looping to complex artificial intelligence responses.
All these computational techniques are impossible in
acoustic instruments and their theoretical implica-
tions unavoidably involve an explicit systemic repre-
sentation of music as a rule-based field or a creative
search space (Boden 1990).
I will define the computational music system as an
epistemic tool, as an instrument (organon) whose
design, practice and often use are primarily sym-
bolic.
1
The concept of epistemic tools is not intended
to define exclusively digital technologies; it includes
all tools that work as props for symbolic offloading
in our cognitive pro cess. Good examples of symbolic
but analogue machines are the abacus and the astro-
labe. In order to clarify this point, this paper proceeds
to explore the nature of embodiment as a non-
representational or non-sym bolic process. It then
looks at how human cognition can make use of
material supports outside the body, echoing Andy
Clark’s statement that cognitive processes can ‘extend
outside the head of an individual’ (Clark 1996: 81).
The central question then becomes what it means
when music is composed and performed with intelli-
gent artefacts that are inscribed with a specific music
theoretical outlook.
The question of epistemic tools can be posed like
this: if much of our thinking happens ‘in the wild’
(Hutchins 1995), external to our body, as a socio-
cultural process that uses technology as an external
scaffolding of cognitive processes, and if our learning is
largely a process of incorporating knowledge in a non-
symbolic way through an enactive relationship with
our tools and environment; how do computers, as
necessarily symbolic devices, enable, produce, main-
tain, support, augment but also constrain and limit our
cognitive processes and therefore creative output?
The aim of this paper is to identify two linked
distinctions that are becoming increasingly apparent
in the design and analysis of modern musical instru-
ments. Firstly, it examines the different embodied
experiences available in acoustic and digital in-
struments (which is primarily a phenomenological
investigation); this route has been navigated most
often in the NIME community. Secondly, it explores
the disparate theoretic and material affordances of
acoustic and digital instruments (here seen as an
epistemological enquiry); whilst related to and con-
straining the first area, previous work has largely
neglected this perspective. This paper therefore seeks
to correct the imbalance between the two identified
distinctions in the analysis of digital musical instru-
ments by addressing the epistemological nature of
our new musical instrument s.
2. THE MASTERY OF A MUSICAL
INSTRUMENT: A SUB-SYMBOLIC SKILL
ACQUISITION
Twentieth-century cognitivism has not been successful
in portraying human intelligence, and various
approaches have emerged that propose different views
of cognition where typically the body and the environ-
ment enter the equation.
2
One of the most musically
relevant approaches is enactivism, as developed by
Varela, Thompson and Rosch (1991). This theory, with
roots in biology but inspired by phenomenology and
Eastern thought, depicts the mind as necessarily
embodied in an external environment. In enactivism
there is no distinction between perception and action;
both co-emerge through the agent’s active involvement
with the world. This involvement is primarily non-
representational – in other words, it is at a level where
the cognition is subconscious, pre-conceptual, dis-
tributive and emergent. Varela et al. define the term:
We propose the term enactive to emphasize the growing
conviction that cognition is not the representation of a
pregiven world by a pregiven mind but is rather the
1
As an example we might take Michel Waisvisz’s The Hands
instrument. Every sensor of the complex interface is mapped to a
specific parameter in the software-based sound engine. A change in
the engine will result in a new (or altered) instrument. Although the
interface has not been altered by a change in the mapping algo-
rithm, the instrument behaves differently. For Waisvisz, changing
the algorithms that constitute the sound engine means learning a
new instrument, which involves the re-incorporation of the con-
ceptual understanding of the engine’s functionality into bodily
memory (Waisvisz 1999, 2005).
2
Cognitivism is here used as a term that denotes the trend in cog-
nitive science from the mid twentieth century to view human cog-
nition as primarily symbolic. This resulted in a tradition in
Artificial Intelligence now called GOFAI (Good Old Fashioned
AI). Arguably, the distinct fields of cognitive science and computer
technology symbiotically constituted each other’s rise through this
period. The new theories criticising cognitivism include: Con-
nectionism, where the idea is to build artificial neural networks that
perform cognitive tasks through non-representational content
(McClelland, Rumelhart and the PDP Research Group 1986);
Enactivism, which claims that the whole body and the environment
becomes part of the cognitive function (Varela et al. 1991);
Dynamic Systems, a non-representational theory claiming that
cognition (and consciousness) arises as epi-phenomena of the
process of being in the world (Brooks 1991); Situated Cognition,a
theory of knowledge acquisition as situated, being in part a product
of the activity, context and culture in which it is developed and
used (Brown et al. 1989); Situated Action, an emphasis on the
constitutional context of all action as emergence (Suchman 1987);
Activity Theory, where the focus is on human tool use and the
cultural and technological mediation of human activity (Bertelsen
and Bødker 2003); Embodied Cognition, where thinking and acting
is seen as one process, emphasising our situatedness, and claiming
that our cognitive system emerges from interaction with the world
(Anderson 2003); and Distributed Cognition, where cognition is
seen as an interaction between human and artefacts, and emphasis
is laid on the social nature of human existence (Hutchins 1995).
Of Epistemic Tools 169
http://journals.cambridge.org Downloaded: 01 Jul 2009 IP address: 139.184.30.131
enactment of a world and a mind on the basis of a
history of the variety of actions that a being in the world
performs. (Varela et al. 1991: 9)
Importantly, this does not mean that humans are
not symbolic cognisers. That fact is obvious by
looking at how humans constantly create and use
symbolic systems such as language, mathematics and
music, or games such as chess. Stressing that they ‘see
symbols as a higher-level description of properties
that are ultimately embedded in an underlying dis-
tributed system’ (Varela et al. 1991: 101), Varela et al.
find it more appropriate to define symbols (or cog-
nitive representations) as ‘approximate macrolevel
descriptions of operations whose governing principles
reside at a subsymbolic level’ (Varela et al. 1991: 102).
Varela et al. describe embodiment as an enactive
stance towards the world where we create our (not ‘the’)
world through an active engagement with it. Enactivism
is resourceful in explanations of embodiment, some-
thing we should bear in mind when analysing how
acoustic musicians incorporate knowledge of their
instruments through repeated practice. It also explains
how symbolic systems are higher-level descriptions of
phenomena understood by the agent through bodily
perceptions. There is a general consensus in cognitive
science that musicians (or athletes for that matter) learn
their skills gradually through persistent practice and a
minimum of verbal instructions (Dreyfus and Dreyfus
1986). Such bodily incorporation transforms our tools
into ready-at-hand phenomena (Heidegger 1962) that
are not based on symbolic or theoretical relations any
more. The focus becomes the act and not the object, or,
as Winograd and Flores explain in a Heideggerian
manner, ‘[m]y ability to act comes from my familiarity
with hammering, not my knowledge of a hammer’
(Winograd and Flores 1986: 33).
According to the enactive view, the skill acqu isition
related to learning an acoustic instrument is highly
embodied, non-symbolic and perceptuo-motor
based.
3
It explains how Ihde’s ‘embodiment relations’
to the world are established, and provides a descrip-
tion of this process of incorporation
4
from the level of
biology. An important research question here
becomes to explore, together with the enquiry into
the epistemological or music theoretical nature of
digital tools , how the digital musical instrument
manifests a different relationship between the human
body and the body of the instrument itself through its
characteristic split between the interface and the
sound engine.
3. THE EXTENDED MIND
In the 1990s, working in another strand of cognitive
science from that of Varela et al., Andy Clark
developed his theory of the ‘ext ended mind’. Clark
illustrates how people use props in the environment
to extend their cognitive capacity and ease cognitive
load. Sticky notes, notebooks, diagrams, models, and
so on all serve as scaffoldings onto which we ‘offload’
our cognition. It should be noted that this is not some
kind of mystical panpsychism, as Clark and Chalmers
adamantly point out that they do not equate the
cognitive process with consciousness (Clark and
Chalmers 1998). The cognitive process happens both
inside and outside the skull, an observation reminis-
cent of Wittgenstein’s account of thinking as physical
activity (Wittgenstein 1969: 6).
[In certain conditions, t]he human organism is linked
with an external entity in a two-way interaction, creating
a coupled system that can be seen as a cognitive system in
its own right. All the components in the system play an
active causal role, and they jointly govern behavior in
the same sort of way that cognition usually does. If we
remove the external component the system’s behavioral
competence will drop, just as it would if we removed
part of its brain. Our thesis is that this sort of coupled
process counts equally well as a cognitive process,
whether or not it is wholly in the head. (Clark and
Chalmers 1998: 7)
Objects and artefacts serve as an external playground
for thinking. We calculate on paper, draw graphs,
build models, make reminders of all kinds, and, as
Clark illustrates, we reorgan ise the pieces when
playing Scrabble. According to this view, it does not
matter whether information is stored and processed
in the brain or outside in the environment, what
matters is how the data is retrieved and taken into use
(Clark 2003).
While the theory of extended mind has much
potential for interesting discussions in cognitive sci-
ence, it is pertinent, in the context of musical tech-
nology, to probe deeper. Indeed, the picture is more
intricate since we are surrounded by objects and
technologies (or ‘thrown’ into world of equipment as
Heidegger would describe it) that contain complex
material properties, scripts of their usage (Latour
1994), and even politics (Winner 1980). The objects
around us – the technologies that serve as props in
our thinking and music making – are stuffed with
people and their ideas; in them we find programmes
of action, manuals of behaviour, and political and
sociocultural constructions, including aesthetic ten-
dencies. Technological objects are therefore never
neutral, they contain scripts that we subscribe to or
reject according to our ideological constitution. The
problem is that the scripts are often well hidden and
concealed, which can resul t in an uncritical use of
3
Useful accounts of the non-symbolic nature of learning are David
Sudnow’s ethnographic account of learning to play the jazz piano
(Sudnow 2001) or Dreyfus and Dreyfus’ analysis of the five stages
of skill acquisition (Dreyfus and Dreyfus 1986).
4
On the process of incorporation in the context of embodiment, see
Hayles (1999).
170 Thor Magnusson
http://journals.cambridge.org Downloaded: 01 Jul 2009 IP address: 139.184.30.131
creative technologies – technologies inherent with
ideological content.
To illustrate this point in the field of musical
instruments, we see that the piano keyboard ‘tells us’
that microtonality is of little importance (and much
Western music theory has wholly subscribed to that
script); the drum-sequencer that 4/4 rhythms and
semiquavers are more natural than other types; and
the digital audio workstation, through its affordances
of copying, pasting and looping, assures us that it
is perfectly normal to repeat the same short perfor-
mance over and over in the same track. It is therefore
germane to ask ourselves how much theory is
inscribed into our new tools, particularly with regard
to how automated, struc turally complex and aiding
(in composition and performance) our new musical
instruments have become.
4. MATERIAL EPISTEMOLOGIES
Things are impregnated with thoughts; they are
embedded with ideologies that have ethical, political
and aesthetical implications. Furthermore, we use
those artefacts in our daily tasks as extensions of our
cognitive mechanism. But how do things ‘contain’
knowledge? How do we write our knowledge into
artefacts, and how do we read that knowledge from
them? By the same token, how does this relate to
digital musical instruments?
Davis Baird (2004) shows how material objects can
have a different epistemological status to statements
of language. Technological objects may contain
functions which their users unde rstand, but woul d
not be able to describe in language as ‘[t]he material
products a re constitutive of scientific knowledge in a
manner different from theory, and not simply
‘‘instrumental to’’ theory’ (Baird 2004: 1). Much like
Heidegger and Ihde, Baird shows that technological
objects can precede science, and afford scientific dis-
coveries through their physical structure and func-
tionality. The point here is not what precedes what,
but rather that the instrument becomes an expression
in itself, an externalisation of knowledge in a form
that is not symbolic but material. Additionally, it is
not only designed objects that inhere knowledge, but
natural objects as well, by means of the physical and
mechanistic properties of their material.
Knowledge can be expressed in many ways. Theories
express knowledge through the descriptive and argu-
mentative functions of language. Instruments express
knowledge both through the representational possibi-
lities that materials offer and through the instrumental
functions they deploy. (Baird 2004: 131)
Baird is well aware of Bruno Latour’s ideas of
concretisation: namely, that when many elements
combine into one actor and start to operate as a unity
(with either human or non-human agency), they
gradually become a black box. Whe n the black box
works, its origins are forgotten and thus ‘para-
doxically, the more science and technology succeed,
the more opa que and obscure they become’ (Latour
1999: 304). Baird agrees with Latour on the nature of
scientific blackboxing, but highlights another and
perhaps more epistemologically active function of the
black box itself: while talking about a particular
instrument, called Spectromet, Baird says
[t]he knowledge used in this context is tacit in the sense
that those using the instrument (typically) could not
articulate the understanding of spectrochemestry they
deploy in doing so. Nonetheless, they can use it. This
spectrochemical knowledge has become detached. It has
gone inside – inside the instrument – and can now tacitly
serve other technical and scientific purposes. (Baird
2004: 163)
Here (and spectacularly exemplified in the digital
musical system) we see how the blackboxed instrument
contains the knowledge of its inventors, which means
that the users of the instrument do not need to have a
deep knowledge of its internal functions. If we assume
that both the designers and the users of the instrument
have an understanding of it, this understanding is very
different and attained from distinct origins. The for-
mer creates the instrument from a conceptual under-
standing of the domain encapsulated by it, whereas
the latter gains operational knowledge that emerges
through use (or habituation) and not from abstract
understanding of the internal functionality. This pic-
ture is particularly complex in today’s new musical
interfaces as typically their designers are also the per-
formers. This implies a continuous oscillation between
a mode of conceptual (system design) engagement with
the instrument and embodied (performative) relation-
ship with it. Again, we are reminded of Waisvisz’s
stance introduced above, where we might talk of two
modes: that of the instrument designer and the
instrument player.
5. THE ACOUSTIC, THE ELECTRIC
AND THE DIGITAL: INTERFACES OF
A DIFFERENT KIND
When the technological artefact is made of material
substrata (as opposed to symbolic), it can contain
knowledge that precedes the scientific understanding
of its functioning. The acoustic instrument is a good
example. The so phisticated sound of the clarinet or
the cello was developed over an extended period of
time, but this sound was mature long before Fourier
or Helmholtz brought forth their theories of sinu-
soidal functions and timbre. Strings, wood and brass
tacitly encompass the theories of sound in their
materiality. The observat ion here is that acoustic
Of Epistemic Tools 171