Title: Distinct representational structure and localization for visual encoding and
1
recall during visual imagery
2
3
4
Authors: Wilma A. Bainbridge (1,2), Elizabeth H. Hall (3,2), Chris I. Baker (2)
5
6
Affiliations:
7
1 – Department of Psychology, University of Chicago; Chicago, IL 60637
8
2 – Laboratory of Brain and Cognition, National Institute of Mental Health; Bethesda, MD 20814
9
3 – Department of Psychology, University of California Davis; Davis, CA 95616
10
11
Corresponding Author: Wilma A. Bainbridge, wilma@uchicago.edu, (773)-702-3189
12
5848 S University Ave, Beecher Hall 303, Chicago, IL 60637
13
14
Running title: Distinct representations for encoding and recall
15
16
made available for use under a CC0 license.
certified by peer review) is the author/funder. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also
The copyright holder for this preprint (which was notthis version posted October 12, 2020. ; https://doi.org/10.1101/842120doi: bioRxiv preprint
Abstract
17
During memory recall and visual imagery, reinstatement is thought to occur as an
18
echoing of the neural patterns during encoding. However, the precise information in these
19
recall traces is relatively unknown, with previous work primarily investigating either broad
20
distinctions or specific images, rarely bridging these levels of information. Using ultra-high-field
21
(7T) fMRI with an item-based visual recall task, we conducted an in-depth comparison of
22
encoding and recall along a spectrum of granularity, from coarse (scenes, objects) to mid (e.g.,
23
natural, manmade scenes) to fine (e.g., living room, cupcake) levels. In the scanner, participants
24
viewed a trial-unique item, and after a distractor task, visually imagined the initial item. During
25
encoding, we observed decodable information at all levels of granularity in category-selective
26
visual cortex. In contrast, information during recall was primarily at the coarse level with fine
27
level information in some areas; there was no evidence of mid-level information. A closer look
28
revealed segregation between voxels showing the strongest effects during encoding and those
29
during recall, and peaks of encoding-recall similarity extended anterior to category-selective
30
cortex. Collectively, these results suggest visual recall is not merely a reactivation of encoding
31
patterns, displaying a different representational structure and localization from encoding,
32
despite some overlap.
33
34
Keywords: 7T fMRI, encoding-recall similarity, objects, representational similarity analyses,
35
scenes
36
37
38
made available for use under a CC0 license.
certified by peer review) is the author/funder. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also
The copyright holder for this preprint (which was notthis version posted October 12, 2020. ; https://doi.org/10.1101/842120doi: bioRxiv preprint
Introduction
39
When we visually recall an object or scene, our memory contains rich object and spatial
40
information (Bainbridge et al. 2019). During such recollection, our brain is thought to reinstate
41
neural patterns elicited by the initial perception (McClelland et al. 1995; Buckner and Wheeler
42
2001; Tompary et al. 2016; Dijkstra et al 2019). One common view is that the hippocampus
43
indexes populations of neocortical neurons associated with that memory (Teyler and Rudy
44
2007; Danker and Anderson 2010; Schultz et al. 2019). Under this view, representations in
45
hippocampus are largely independent of a memory’s perceptual content (Davachi 2006; Liang
46
et al. 2013; Huffman and Stark 2014). In contrast, the neocortex is thought to show sensory
47
reinstatement, where the same regions show the same representations during recall as during
48
encoding (Wheeler et al. 2000; Kahn et al. 2004; Staresina et al. 2012; Ritchey et al. 2013; Lee et
49
al. 2012; O’Craven and Kanwisher 2000; Dijkstra et al. 2017). However, prior work has focused
50
on specific levels of information (e.g. broad stimulus class, specific image) and the extent to
51
which representations during recall reflect the same information as during perception, at all
52
levels of granularity (from individual exemplar up to broad stimulus category), is unclear. Here,
53
using ultra-high-field (7T) fMRI, we conducted an in-depth investigation of the content of
54
encoded and recalled representations of objects and scenes across cortex, hippocampus, and
55
the medial temporal lobe, assessing the granularity of detail in the representations of individual
56
items.
57
First, we employed a hierarchically organized stimulus set (Figure 1a) with three levels
58
of granularity from coarse (scenes/objects) to mid (e.g., natural/manmade scenes) to fine (e.g.,
59
bedrooms/conference rooms) level. Prior work comparing encoding and recall have primarily
60
made available for use under a CC0 license.
certified by peer review) is the author/funder. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also
The copyright holder for this preprint (which was notthis version posted October 12, 2020. ; https://doi.org/10.1101/842120doi: bioRxiv preprint
investigated memory content at opposite ends of this granularity spectrum. At a coarse level,
61
recall of stimulus classes (faces, scenes, objects) have been reported to reactivate high-level
62
visual regions (Polyn et al. 2005; Johnson et al. 2009; Reddy et al. 2010; LaRocque et al. 2013)
63
and produce differentiable responses in hippocampus (Ross et al 2018). At the fine level, other
64
work has shown reinstatement for individual images, with specific visual stimuli decodable in
65
high-level visual cortex (Dickerson et al. 2007; Buchsbaum et al. 2012; Lee et al. 2012; Kuhl and
66
Chun 2014) and medial temporal lobe (Zeineh et al. 2003; Gelbard-Sagiv et al. 2008; Chadwick
67
et al. 2010; Wing et al. 2015; Mack and Preston 2016; Tompary et al. 2016; Lee et al. 2019).
68
Decoding for specific images (Thirion et al. 2006; Naselaris et al. 2015), positions (Stokes et al.
69
2011) and orientations (Klein et al. 2004; Albers et al. 2013) is even present in early visual
70
cortex during visual imagery. However, it is often unclear what information is driving
71
discrimination across the brain: fine-level image-specific information, coarse-level perceptual
72
category information, or information unrelated to stimulus content such as memory strength.
73
For example, while recalled grating orientation is decodable from early visual cortex (V1-V3),
74
reinstatement strength but not content is decodable from the hippocampus (Bosch et al. 2014).
75
Further, few studies have investigated the ability to detect reinstatement of mid-level
76
information (e.g., is it a natural or manmade scene, a big or small object) during recall, even
77
though such information is known to be decodable during perception (e.g., Park et al. 2011;
78
Kravitz et al. 2011; Konkle et al. 2012). Our approach using nested levels of stimulus
79
information reveals what granularity of information is contained in regions across the visual
80
processing pathway, and whether reinstatement is simply an echo of the same response from
81
encoding to recall.
82
made available for use under a CC0 license.
certified by peer review) is the author/funder. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also
The copyright holder for this preprint (which was notthis version posted October 12, 2020. ; https://doi.org/10.1101/842120doi: bioRxiv preprint
Second, to isolate the activity specific to recall, we adopted a visual imagery task
83
focusing on recall of individual items without requiring the learning of cue-stimulus
84
associations, which have commonly been used (e.g., Ganis et al. 2004; Kuhl et al. 2012; Zeidman
85
et al. 2015a; Jonker et al. 2018). Recalled representations in associative tasks are likely to
86
contain information not only about the recalled item, but also the cue and the association itself.
87
Further, there are differences in neocortex when performing an associative versus item-based
88
memory task (Staresina and Davachi 2006). In fact, the neural representation of a target may be
89
largely dependent on what cue it is associated with (Xiao et al. 2017). Here, we employ an item-
90
based recall task in which participants encode trial-unique images, and following a distractor
91
task, recall that specific image. This approach allows us to investigate the recall of individual
92
items, without the learning of associations.
93
Using this direct recall task and nested stimulus structure, we find striking differences in
94
the representational structure and spatial localization for visual encoding and recall, suggesting
95
recall patterns are not just a repetition of patterns during encoding, despite some similarities.
96
97
Materials and Methods
98
Participants
99
Thirty-four adults were recruited for the experiment. All participants were healthy,
100
right-handed, and had corrected or normal vision. Twelve participants were unable to complete
101
the experiment due to discomfort in the 7T scanner, drowsiness, or scanner malfunction, and
102
their data were excluded from the study. This level of participant dropout is not unusual for 7T
103
scans, given that nausea and vertigo occasionally occur, and the bore is more restrictive than
104
made available for use under a CC0 license.
certified by peer review) is the author/funder. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also
The copyright holder for this preprint (which was notthis version posted October 12, 2020. ; https://doi.org/10.1101/842120doi: bioRxiv preprint