Pickering, M.J., and Garrod, S. (2013) An integrated theory of language
production and comprehension. Behavioral and Brain Sciences, 36 (04). pp.
329-347. ISSN 0140-525X
Copyright © 2013 Cambridge University Press
http://eprints.gla.ac.uk/82858/
Deposited on:
02 September 2014
Enlighten – Research publications by members of the University of Glasgow
http://eprints.gla.ac.uk
An integrated theory of language
production and comprehension
Martin J. Pickering
Department of Psychology, University of Edinburgh, Edinburgh EH8 9JZ,
United Kingdom
martin.pickering@ed.ac.uk
http://www.ppls.ed.ac.uk/people/martin-pickering
Simon Garrod
University of Glasgow, Institute of Neuroscience and Psychology,
Glasgow G12 8QT, United Kingdom
simon@psy.gla.ac.uk
http://staff.psy.gla.ac.uk/∼simon/
Abstract: Currently, production and comprehension are regarded as quite distinct in accounts of language processing. In rejecting this
dichotomy, we instead assert that producing and understanding are interwoven, and that this interweaving is what enables people to
predict themselves and each other. We start by noting that production and comprehension are forms of action and action perception. We
then consider the evidence for interweaving in action, action perception, and joint action, and explain such evidence in terms of prediction.
Specifically, we assume that actors construct forward models of their actions before they execute those actions, and that perceivers of
others’ actions covertly imitate those actions, then construct forward models of those actions. We use these accounts of action, action
perception, and joint action to develop accounts of production, comprehension, and interactive language. Importantly, they incorporate
well-defined levels of linguistic representation (such as semantics, syntax, and phonology). We show (a) how speakers and comprehenders
use covert imitation and forward modeling to make predictions at these levels of representation, (b) how they interweave production and
comprehension processes, and (c) how they use these predictions to monitor the upcoming utterances. We show how these accounts
explain a range of behavioral and neuroscientific data on language processing and discuss some of the implications of our proposal.
Keywords: comprehension; covert imitation; dialogue; forward model; language; prediction; production
1. Introduction
Current accounts of language processing treat production and
comprehension as quite distinct from each other. The split is
clearly reflected in the structure of recent handbooks and
textbooks concerned with the psychology of language (e.g.,
Gaskell 2007;Harley2008). This structure does not merely
reflect organizational convenience but instead treats compre-
hension and production as two different questions to investi-
gate. For example, researchers assume that the processes
involved in comprehending a spoken or written sentence,
such as resolving ambiguity, may be quite distinct from the
processes involved in producing a description of a scene. In
neurolinguistics, the “classic” Lichtheim–Broca–Wernicke
model assumes distinct anatomical pathways associated with
production and comprehension, primarily on the basis of
deficit–lesion correlations in aphasia (see Ben Shalom &
Poeppel 2008). This target article rejects such a dichotomy.
In its place, we propose that producing and understanding
are tightly interwoven, and this interweaving underlies
people’s ability to predict themselves and each other.
1.1. The traditional independence of production and
comprehension
To see the effects of the split, we need to think about
language use both within and between individuals, in
terms of a model of communication (Fig. 1).
This model includes “thick” arrows between message
and (linguistic) form, corresponding to production and
comprehension. The prod uction arrows represent the fact
that production may involve converting one message into
form (serial account) or the processor may convert multiple
messages at once, then select one (parallel account). Within
production, the “internal” arrows signify feedback (e.g.,
from phonology to syntax), which occurs in interactive
accounts but not purely feedforward accounts. Note that
these arrows are consistent with any type of information
(linguistic or nonlinguistic) being used during production.
The arrows play an an alogous role within comprehension
(e.g., the internal arrows could signify feedback from
semantics to syntax). In contrast, the arrows corresponding
to sound are “thin” because a single sequence of sounds is
sent forward between the speakers. If communication is
fully successful, then A’s message
1
=B’s message
1
. Similarly,
there is a “ thin” arrow for thinking because such accounts
assume that each individual converts a single message
(e.g., an understanding of a question, message
1
) into
another (e.g., an answer, message
2
), and the answer does
not affect the understanding of the question.
The model is split vertically between the processes in
different individuals, who of course have independent
minds. But it is also split horizontally, because the pro-
cesses underlying production and comprehension within
each individual are separated. The traditional model
assumes discrete stages: one in which A is producing and
BEHAVIORAL AND BRAIN SCIENCES (2013) 36, 329–392
doi:10.1017/S0140525X12001495
© Cambridge University Press 2013 0140-525X/13 $40.00 329
B is comprehending an utterance, and one in which B is
producing and A is comprehending an utterance. Each
speaker constructs a message that is translated into sound
before the addressee responds with a new message.
Hence, dialogue is “serial monologue,” in which interlocu-
tors alternate between production and comprehension.
In conversation, however, interlocutors’ contributions
often overlap, with the addressee providing verbal or non-
verbal feedback to the speaker, and the speaker altering
her contribution on the basis of this feedback. In fact, such
feedback can dramatically affect both the quality of the
speaker’s contribution (e.g., Bavelas et al. 2000)andthe
addressee’s understanding (Schober & Clark 1989). This of
course means that both interlocutors must simultaneously
produce their own contributions and comprehend the
other’s contribution. Clearly, an approach to language pro-
cessing that assumes a temporal separation between pro-
duction and comprehension cannot explain such behavior.
Interlocutors are not static, as the traditional model
assumes, but are “moving targets” performing a joint
activity (Garrod & Pickering 2009). They do not simply
transmit messages to each other in turn but rather nego-
tiate the form and meaning of expressions they use by inter-
weaving their contributions (Clark 1996), as illustrated in
(1a–1c), below (from Gregoromichelaki et al. 2011). In
(1b), B begins to ask a question, but A’s interruption (1c)
completes the question and answers it. B, therefore, does
not discretely encode a complete message into sound but,
rather, B and A jointly encode the message across (1b–c).
1a—–A: I’m afraid I burnt the kitchen ceiling
1b—–B: But have you
1c—–A: burned myself? Fortunately not.
The horizo ntal split is also challenged by findings from
isolated instances of comprehension or production. Take
picture-word interference, in which participants are told
to name a picture (e.g., of a dog) while ignoring a spoken
or written distractor word (e.g., Schriefers et al. 1990). At
certain timings, they are faster naming the picture if the
word is phonologically related to it (dot) than if it is not.
The effect cannot be caused by the speaker’s interpreting
dot before producing dog – the meaning of dot is not the
cause of the facilitation. Rather, the participant accesses
phonology during the comprehension of dot, and this
affects the construction of phonology during the pro-
duction of dog. So experiments such as these suggest that
production and comprehension are tightly interwoven.
Quite ironically, most psycholinguistic theories attempt to
explain either production or comprehension, but a great
many experiments appear to involve both. Single word
naming is typically used to explain comprehension but
involves production (see Bock 1996). Sentence completion
is often used to explain production but involves compre-
hension (e.g., Bock & Miller 1991). Similarly, the finding
that word identification can be affected by externally con-
trolled cheek movement (Ito et al. 2009) suggests that pro-
duction in fluences comprehension.
In addition, production and comprehension appear to
recruit strongly overlapping neural circuits (Scott & Johns-
rude 2003; Wilson et al. 2004). For example, Paus et al.
(1996) found activation (dependent on the rate of
speech) of regions associated with speech perception
when people whispered but could not hear their own
speech. Listeners also activate appropriate muscles in the
tongue and lips while listening to speech but not nonspeech
(Fadiga et al. 2002; Watkins et al. 2003). Additionally,
increased muscle activity in the lips is associated with
increased activity (i.e., blood flow) in Broca’s area,
suggesting that this area mediates between the comprehen-
sion and production systems during speech perception
(Watkins & Paus 2004). There is also activation of brain
areas associated with production during aspects of
comprehension from phonology (Heim et al. 2003) to nar-
rative structure (Mar 2004); see Scott et al. (2009) and
Pulvermüller and Fadiga (2010). Finally, Menenti et al.
(2011) found massive overlap between speaking and listen-
ing for regions showing functional magnetic resonance
imaging (fMRI) adaptation effects associated with repeat-
ing language at different linguistic levels (see also Segaert
et al. 2012). These results are inconsistent with separation
of neural pathways for production and comprehension in
the classical Lichtheim–Broca–Wernicke neurolinguistic
model.
In conclusion, the evidence from dialogue, psycholin-
guistics, and cognitive neuroscience all casts doubt on the
independence of production and comprehension, and
therefore on the horizontal split assumed in Figure 1. Let
us now ad dress two theoretical issues relating to the aban-
donment of this split, and then ask what kind of model is
compatible with the interweaving of production and
comprehension.
1.2. Modularity and the cognitive sandwich
Much of psycholinguistics has sought to test the claim that
language processing is modular (Fodor 1983). Such
accounts investigate the way in which information travels
MARTIN J. PICKERING is Professor of the Psychology of
Language and Communication at the University of
Edinburgh. He is the author of more than 100 journal
articles and numerous other publications in language
production, comprehension, dialogue, reading, and
bilingualism. His articles have appeared in Psychological
Bulletin; Trends in Cognitive Sciences; Psychological
Science; Cognitive Psychology; Journal of Memory and
Language; Cognition; Journal of Experimental Psychol-
ogy: Learning, Memory, and Cognition; and many other
journals. In particular, he published “Toward a mechan-
istic psychology of dialogue ” (Behavioral and Brain
Sciences, 2004) with Simon Garrod. He is a Fellow of
the Royal Society of Edinburgh and the editor of
Journal of Memory and Language.
S
IMON GARROD is Professor of Cognitive Psychology at
the University of Glasgow. Between 1989 and 1999 he
was also Deputy Director of the ESRC Human Com-
munication Research Centre. He has published two
books, one with Anthony J. Sanford, Understanding
written language, and one with Kenny R. Coventry,
Seeing, saying and acting: The psychological semantics
of spatial prepositions. Additionally, he has published
more than 100 research papers on various aspects of
the psychology of language. His special interests
include discourse processing, language processing in
dialogue, psychological semantics, and graphical com-
munication. He is a Fellow of the Royal Society of
Edinburgh.
Pickering & Garrod: An integrated theory of language production and comprehension
330
BEHAVIORAL AND BRAIN SCIENCES (2013) 36:4
between the boxes in a model such as in Figure 1. In par-
ticular, the arrows labeled thinking correspond to “central
processes” and contain representations in some kind of
language of thought. Researchers are particularly con-
cerned with the extent to which thinking arrows are separ-
ated from the production and comprehension arrows.
Modular theories assume that some aspects of production
or comprehension do not make reference to “central pro-
cesses” (e.g., Frazier 1987 ; Levelt et al. 1999). In contrast,
interactionist theories allow “central processes” to directly
affect production or comprehension (e.g., Dell 1986; Mac-
Donald et al. 1994; Trueswell et al. 1994). But both types of
theory maintain that production and comprehension are
separated from each other. In this sense, both types of
theory are modular and are compatible with Figure 1.
In fact, Hurley (2008a) argued that traditional cognitive
psychology assumes this type of modularity in order to keep
action and perception separate. She referred to this
assumption as the cognitive sandwich. Individuals perceive
the world, reason about their perceptions using thinking
(i.e., cognition), and act on the basis of those thoughts.
Researchers assume that action and perception involve sep-
arate representations and processes and study one or the
other but not both (and they are kept separate in textbooks
and the like). In Hurley’s terms, the cognitive “meat ” keeps
the motor “bread” separate from the perceptual “bread.”
1
She argued that perception and action are interwoven
and, therefore, rejected the cognitive sandwich.
Importantly, language production is a form of action and
language comprehension is a form of perception. Therefore,
traditional psycholinguistics also assumes the cognitive sand-
wich, with the thinking “meat” keepingaparttheproduction
and comprehension “bread.” But if action and perception
are interwoven, then production and comprehension are
interwoven as well, and so accounts of language processing
should also reject the cognitive sandwich.
1.3. Production and comprehension processes
How can production and comprehension both be involved
in isolated speaking or listening? Within the individual, we
mean that production and comprehension processes are
interwoven. Production processes must of course be
used when individuals produce language, and comprehen-
sion processes must be used when they comprehend
language. However, production processes must also be
used during, for example, silent naming, when no utterance
is produced. Silent naming therefore involves some pro-
duction processes (e.g., those associated with aspects of for-
mulation such as name retrieval) but not others (e.g., those
associated with articulation; see Levelt 1989). Likewise,
comprehension processes must occur when a participant
retrieves the phonology of a masked prime word but not
its semantics (e.g., Van den Bussche et al. 2009). And so it
is also possible that production processes are used during
comprehension and comprehension processes used during
production.
How can we distinguish production processes from com-
prehension processes? For this, we assume that (1) people
represent linguistic information at different levels; (2) these
levels are semantics, syntax, and phonology
2
; (3) they are
ordered “higher” to “lower, ” so that a speaker’s message
is linked to semantics, semantics to syntax, syntax to pho-
nology, and phonology to speech. We then assume that a
producer goes from message to sound via each of these
levels (message → semantics → syntax → phonology →
sound), and a comprehender goes from sound to message
in the opposite direction. Given this framework, we
Figure 1. A traditional model of communication between A and B. (comp: comprehension; prod: production)
Pickering & Garrod: An integrated theory of language production and comprehension
BEHAVIORAL AND BRAIN SCIENCES (2013) 36:4 331
define a production process as a process that maps from a
“higher” to a “lower” linguistic level (e.g., syntax to phonol-
ogy) and a comprehension process as a process that maps
from a “lower” to a “higher” level.
3
This means that produ-
cing utterances must involve production processes, but can
also involve comprehension processes; similarly, compre-
hending utterances must involve comprehension processes,
but can also involve production processes.
One possibility is that people have separate production
and comprehension systems. On this account, producing
utterances may make use of feedback mechanisms that
are similar in some respects to the mechanisms of compre-
hension, and comprehending utterances may make use of
feedback mechanisms that are similar in some respects to
the mechanisms of production. This is the position
assumed by traditional interactive models of production
(e.g., Dell 1986) and comprehension (e.g., MacDonald
et al. 1994). In such accounts, production and comprehen-
sion are internally nonm odular, but are modular with
respect to each other. They do not take advantage of the
comprehension system in production or the production
system in comprehension (even though the other system
is often lying dormant).
Very little work in comprehension makes reference to
production processes, with classic theories of lexical proces-
sing (from, e.g., Marslen-Wilson & Welsh 1978 or Swinney
1979 onward) and sentence processing (e.g., Frazier 1987;
MacDonald et al. 1994) making no reference to production
processes (see Bock 1996 for discussion, and Federmeier
2007 for an exception). In contrast, some theories of pro-
duction do incorporate comprehension processes. Most
notably, Levelt (1989) assumed that speakers monito r
their own speech using comprehension processes. They
can hear their own speech (external self-monitoring), in
which case the speaker comprehends his own utterance
just like another person’s utterance; but they can also
monitor a sound-based representation (internal self-moni-
toring), in which comprehension processes are used to
convert sound to message (see sect. 3.1).
In addition, some computationally sophisticated models
can use production and comprehension processes together
(e.g., Chang et al. 2006), use comprehension to assist in the
process of learning to speak (Plaut & Kello 1999), or
assume that comprehension and production use the same
network of nodes and connections so that feedback pro-
cesses during production are the same as feedforward pro-
cesses during comprehension (MacKay 1982). In addition,
Dell has proposed accounts in which feedbac k during pro-
duction is a component of comprehension (e.g., Dell 1988),
although he has also queried this claim on the basis of neu-
ropsychological evidence (Dell et al. 1997, p. 830); see also
the debate between Rapp and Goldrick (2000; Rapp &
Goldrick 2004) and Roelofs (2004).
But none of these theories incorporate mechanisms of
sentence comprehension (e.g., parsing or lexical ambiguity
resolution) into theories of production. We believe that this
is a consequence of the traditional separation of prod uction
and comprehension (as represented in Fig. 1). In contrast,
we propose that comprehension processes are routinely
accessed at different stages in production, and that pro-
duction processes are routinely accessed at different
stages in comprehension.
The rest of this target article develops an account of
language processing in which processes of production and
comprehension are integrated. We assume that instances
of both production and comprehension involve extensive
use of prediction – determining what you yourself or your
interlocutor is likely to say next. Predicting your own utter-
ance involves comprehension processes as well as pro-
duction processes, and predicting another person’s
utterance involves production processes as well as compre-
hension processes.
As we have noted, production is a form of action, and
comprehension is a form of perception. More specifically,
comprehension is a form of action perception – perception
of other people performing actions. We first consider the
evidence for interweaving in action and action perception,
and we explain such evidence in terms of prediction. We
assume that actors construct forward models of their
actions before they execute those actions, and that percei-
vers of others’ actions construct forward models of others’
actions tha t are based on their own potential actions.
Finally, we apply these accounts to joint action.
We then develop these accounts of action, action percep-
tion, and joint action into accounts of production, compre-
hension, and dialogue. Unlike many other forms of action
and perception, language processing is clearly structured,
incorporating well-defined levels of linguistic represen-
tation such as semantics, syntax, and phonology. Thus,
our accounts also include such structure. We show how
speakers and comprehenders predict the content of levels
of representation by interweaving production and compre-
hension processes. We then explain a range of behavioral
and neuroscientific data on language processing, and
discuss some of the implications of the account.
2. Interweaving in action and action perception
For perception and action to be interwoven, there must be
a direct link between them. If so, there should be much evi-
dence for effects of perception on action, and there is. In
one study, participants’ arm movements showed more var-
iance when they observed another person making a differ-
ent versus the same arm movement (Kilner et al. 2003; see
also Stanley et al. 2007). Conversely, there is good evidence
for effects of action on perception. For example , producing
hand movements can facilitate the concurrent visual dis-
crimination of deviant hand postures (Miall et al. 2006),
and turning a knob can affect the perceived motion of a
perceptually bistable object (Wohlschläger 2000). Such evi-
dence immediately casts doubt on the “sandwich” architec-
ture for percep tion and action.
What purpos e might such a link serve? First, it could
facilitate overt imitation, but overt imitation is not
common in many species (see Prinz 2006). Second, it
could be used postdictively, with action representations
helping perceivers develop a stable memory for a percept
or a detailed understanding of it (e.g., via rehearsal), and
perceptual representations doing the same for actors. But
we propose a third alternative: people compute action rep-
resentations during perception and perception represen-
tations during action to aid prediction of what they are
about to perceive or to do, in a way that allows them to
“get ahead of the game” (see Wilson & Knoblich 2005).
4
To explain this, we turn to the theory of forward modeling,
which was first applied to action but has more recently been
applied to action perception. We interpret the theory in a
Pickering & Garrod: An integrated theory of language production and comprehension
332
BEHAVIORAL AND BRAIN SCIENCES (2013) 36:4
, a sequence of sounds that encodes semantics, syntax, and phonology. Notice that t refers to the time of the production command, not the time at which the representations are computed. In turn, the speaker processes this utterance to create an utterance percept, the perception of a sequence of sounds that encodes semantics, syntax, and phonology. Second, an efference copy of i(t) feeds into the forward production model, a computational device which outputs a predicted utterance. This feeds into the forward comprehension model, which outputs a predicted utterance percept (i.e., of the predicted semantics, syntax, and phonology). The monitor can then compare the utterance percept and the predicted utterance percept at one or more linguistic levels (and therefore performs self-monitoring).](/figures/figure-5-a-model-of-production-using-a-snapshot-of-speaking-s85fwp6s.webp)
![Figure 6. A model of the simulation route to prediction during comprehension in Person A. Everything above the solid line refers to B’s unfolding utterance (and is underlined). A predicts B’s utterance p[sem,syn,phon]B (t+1) (i.e., its upcoming semantics, syntax, and](/figures/figure-6-a-model-of-the-simulation-route-to-prediction-jxmxd3ql.webp)
![Figure 7. A and B predicting B’s forthcoming utterance (with B’s processes and representations underlined). B’s production command iB(t) feeds into B’s production implementer and leads to B’s utterance p[sem,syn,phon]B (t). A covertly imitates B’s utterance and uses A’s forward production model to predict B’s forthcoming utterance (at time t+1). B simultaneously constructs the next production command (the dotted line indicates that this command is causally linked to the previous action command for B but not A) and uses B’s forward production model to predict B’s forthcoming utterance. If A and B are coordinated, then A’s prediction of B’s utterance and B’s prediction of B’s utterance (in the dotted box) should match. Moreover, they may both match B’s forthcoming (actual) utterance at time t+1 (not shown).](/figures/figure-7-a-and-b-predicting-bs-forthcoming-utterance-with-bs-3ke6q4k5.webp)