Showing papers in "Journal of Vision in 2019"

Ocular tracking of occluded ballistic trajectories: Effects of visual context and of target law of motion.

Abstract: In tracking a moving target, the visual context may provide cues for an observer to interpret the causal nature of the target motion and extract features to which the visual system is weakly sensitive, such as target acceleration. This information could be critical when vision of the target is temporarily impeded, requiring visual motion extrapolation processes. Here we investigated how visual context influences ocular tracking of motion either congruent or not with natural gravity. To this end, 28 subjects tracked computer-simulated ballistic trajectories either perturbed in the descending segment with altered gravity effects (0g/2g) or retaining natural-like motion (1g). Shortly after the perturbation (550 ms), targets disappeared for either 450 or 650 ms and became visible again until landing. Target motion occurred with either quasi-realistic pictorial cues or a uniform background, presented in counterbalanced order. We analyzed saccadic and pursuit movements after 0g and 2g target-motion perturbations and for corresponding intervals of unperturbed 1g trajectories, as well as after corresponding occlusions. Moreover, we considered the eye-to-target distance at target reappearance. Tracking parameters differed significantly between scenarios: With a neutral background, eye movements did not depend consistently on target motion, whereas with pictorial background they showed significant dependence, denoting better tracking of accelerated targets. These results suggest that oculomotor control is tuned to realistic properties of the visual scene.

Novel procedure for generating continuous flash suppression: Seurat meets Mondrian.

Abstract: Continuous flash suppression (CFS) entails presentation of a stationary target to one eye and an animated sequence of arrays of geometric figures, the mask, to the other eye. The prototypical CFS sequence comprises different sized rectangles of various colors, dubbed Mondrians. Presented as a rapid, changing sequence to one eye, Mondrians or other similarly constructed textured arrays can abolish awareness of the target viewed by the other eye for many seconds at a time, producing target suppression durations much longer than those associated with conventional binocular rivalry. We have devised an animation technique that replaces meaningless Mondrian figures with recognizable visual objects and scenes as inducers of CFS, allowing explicit manipulation of the visual semantic content of those masks. By converting each image of these CFS sequences into successively presented objects or scenes each comprised of many small, circular patches of color, we create pointillist CFS sequences closely matched in terms of their spatio-temporal power spectra. Randomly rearranging the positions of the pointillist patches scrambles the images so they are no longer recognizable. CFS sequences comprising a stream of different objects produces more robust interocular suppression than do sequences comprising a stream of different scenes, even when the two categories of CFS are matched in root mean square contrast and spatial frequency content. Factors promoting these differences in CFS potency could range from low-level, image-based features to high-level factors including attention and recognizability. At the same time, object- and scene-based CFS sequences, when themselves suppressed from awareness, do not differ in their durations of suppression, implying that semantic content of those images comprising CFS sequences are not registered during suppression. The pointillist technique itself offers a potentially useful means for examining the impact of high-level image meaning on aspects of visual perception other than interocular suppression.

Confidence boosts serial dependence in orientation estimation.

Abstract: In the absence of external feedback, a decision maker must rely on a subjective estimate of their decision accuracy in order to appropriately guide behavior. Normative models of perceptual decision-making relate subjective estimates of internal signal quality (e.g., confidence) directly to the internal signal quality itself, thereby making it unknowable whether the subjective estimate or the underlying signal is what drives behavior. We constructed stimuli that dissociated the human observer's performance on a visual estimation task from their subjective estimates of confidence in their performance, thus violating normative principles. To understand whether confidence influences future decision-making, we examined serial dependence in observer's responses, a phenomenon whereby the estimate of a stimulus on the current trial can be biased toward the stimulus from the previous trial. We found that when decisions were made with high confidence, they conferred stronger biases upon the following trial, suggesting that confidence may enhance serial dependence. Critically, this finding was true also when confidence was experimentally dissociated from task performance, indicating that subjective confidence, independent of signal quality, can amplify serial dependence. These findings demonstrate an effect of confidence on future behavior, independent of task performance, and suggest that perceptual decisions incorporate recent history in an uncertainty-weighted manner, but where the uncertainty carried forward is a subjectively estimated and possibly suboptimal readout of objective sensory uncertainty.

Using deep learning to probe the neural code for images in primary visual cortex

Abstract: Primary visual cortex (V1) is the first stage of cortical image processing, and major effort in systems neuroscience is devoted to understanding how it encodes information about visual stimuli. Within V1, many neurons respond selectively to edges of a given preferred orientation: These are known as either simple or complex cells. Other neurons respond to localized center-surround image features. Still others respond selectively to certain image stimuli, but the specific features that excite them are unknown. Moreover, even for the simple and complex cells-the best-understood V1 neurons-it is challenging to predict how they will respond to natural image stimuli. Thus, there are important gaps in our understanding of how V1 encodes images. To fill this gap, we trained deep convolutional neural networks to predict the firing rates of V1 neurons in response to natural image stimuli, and we find that the predicted firing rates are highly correlated (\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{ u}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\({\overline {{\bf{CC}}} _{{\bf{norm}}}}\) = 0.556 ± 0.01) with the neurons' actual firing rates over a population of 355 neurons. This performance value is quoted for all neurons, with no selection filter. Performance is better for more active neurons: When evaluated only on neurons with mean firing rates above 5 Hz, our predictors achieve correlations of \({\overline {{\bf{CC}}} _{{\bf{norm}}}}\) = 0.69 ± 0.01 with the neurons' true firing rates. We find that the firing rates of both orientation-selective and non-orientation-selective neurons can be predicted with high accuracy. Additionally, we use a variety of models to benchmark performance and find that our convolutional neural-network model makes more accurate predictions.

Challenges to pooling models of crowding: Implications for visual mechanisms.

Abstract: A set of phenomena known as crowding reveal peripheral vision's vulnerability in the face of clutter. Crowding is important both because of its ubiquity, making it relevant for many real-world tasks and stimuli, and because of the window it provides onto mechanisms of visual processing. Here we focus on models of the underlying mechanisms. This review centers on a popular class of models known as pooling models, as well as the phenomenology that appears to challenge a pooling account. Using a candidate high-dimensional pooling model, we gain intuitions about whether a pooling model suffices and reexamine the logic behind the pooling challenges. We show that pooling mechanisms can yield substitution phenomena and therefore predict better performance judging the properties of a set versus a particular item. Pooling models can also exhibit some similarity effects without requiring mechanisms that pool at multiple levels of processing, and without constraining pooling to a particular perceptual group. Moreover, we argue that other similarity effects may in part be due to noncrowding influences like cuing. Unlike low-dimensional straw-man pooling models, high-dimensional pooling preserves rich information about the stimulus, which may be sufficient to support high-level processing. To gain insights into the implications for pooling mechanisms, one needs a candidate high-dimensional pooling model and cannot rely on intuitions from low-dimensional models. Furthermore, to uncover the mechanisms of crowding, experiments need to separate encoding from decision effects. While future work must quantitatively examine all of the challenges to a high-dimensional pooling account, insights from a candidate model allow us to conclude that a high-dimensional pooling mechanism remains viable as a model of the loss of information leading to crowding.

Photoreceptor inputs to pupil control.

Abstract: The size of the pupil depends on light level. Watson and Yellott (2012) developed a unified formula to predict pupil size from luminance, field diameter, age, and number of eyes. Luminance reflects input from the L and M cones in the retina but ignores the contribution of intrinsically photosensitive retinal ganglion cells (ipRGCs) expressing the photopigment melanopsin, which are known to control the size of the pupil. We discuss the role of melanopsin in controlling pupil size by reanalyzing an extant data set (Bouma, 1962). We confirm that melanopsin-weighted quantities, in conjunction with Watson and Yellott's formula, adequately model intensity-dependent pupil size. We discuss the contributions of other photoreceptors into pupil control.

Disentangling bottom-up versus top-down and low-level versus high-level influences on eye movements over time

Abstract: Bottom-up and top-down as well as low-level and high-level factors influence where we fixate when viewing natural scenes. However, the importance of each of these factors and how they interact remains a matter of debate. Here, we disentangle these factors by analyzing their influence over time. For this purpose, we develop a saliency model that is based on the internal representation of a recent early spatial vision model to measure the low-level, bottom-up factor. To measure the influence of high-level, bottom-up features, we use a recent deep neural network-based saliency model. To account for top-down influences, we evaluate the models on two large data sets with different tasks: first, a memorization task and, second, a search task. Our results lend support to a separation of visual scene exploration into three phases: the first saccade, an initial guided exploration characterized by a gradual broadening of the fixation density, and a steady state that is reached after roughly 10 fixations. Saccade-target selection during the initial exploration and in the steady state is related to similar areas of interest, which are better predicted when including high-level features. In the search data set, fixation locations are determined predominantly by top-down processes. In contrast, the first fixation follows a different fixation density and contains a strong central fixation bias. Nonetheless, first fixations are guided strongly by image properties, and as early as 200 ms after image onset, fixations are better predicted by high-level information. We conclude that any low-level, bottom-up factors are mainly limited to the generation of the first saccade. All saccades are better explained when high-level features are considered, and later, this high-level, bottom-up control can be overruled by top-down influences.

A computational-observer model of spatial contrast sensitivity: Effects of wave-front-based optics, cone-mosaic structure, and inference engine.

Abstract: We present a computational-observer model of the human spatial contrast-sensitivity function based on the Image Systems Engineering Toolbox for Biology (ISETBio) simulation framework. We demonstrate that ISETBio-derived contrast-sensitivity functions agree well with ones derived using traditional ideal-observer approaches, when the mosaic, optics, and inference engine are matched. Further simulations extend earlier work by considering more realistic cone mosaics, more recent measurements of human physiological optics, and the effect of varying the inference engine used to link visual representations to psychophysical performance. Relative to earlier calculations, our simulations show that the spatial structure of realistic cone mosaics reduces the upper bounds on performance at low spatial frequencies, whereas realistic optics derived from modern wave-front measurements lead to increased upper bounds at high spatial frequencies. Finally, we demonstrate that the type of inference engine used has a substantial effect on the absolute level of predicted performance. Indeed, the performance gap between an ideal observer with exact knowledge of the relevant signals and human observers is greatly reduced when the inference engine has to learn aspects of the visual task. ISETBio-derived estimates of stimulus representations at various stages along the visual pathway provide a powerful tool for computing the limits of human performance.

Prediction in goal-directed action

Abstract: Prediction allows humans and other animals to prepare for future interactions with their environment. This is important in our dynamically changing world that requires fast and accurate reactions to external events. Knowing when and where an event is likely to occur allows us to plan eye, hand, and body movements that are suitable for the circumstances. Predicting the sensory consequences of such movements helps to differentiate between self-produced and externally generated movements. In this review, we provide a selective overview of experimental studies on predictive mechanisms in human vision for action. We present classic paradigms and novel approaches investigating mechanisms that underlie the prediction of events guiding eye and hand movements.

The role of feature-based attention in visual serial dependence.

Abstract: Perceptual decisions about current sensory input are biased toward input of the recent past-a phenomenon termed serial dependence. Serial dependence may serve to stabilize neural representations in the face of external and internal noise. However, it is unclear under which circumstances previous input attracts subsequent perceptual decisions, and thus whether serial dependence reflects a broad smoothing or selective stabilization operation. Here we investigated whether focusing attention on particular features of the previous stimulus modulates serial dependence. We found an attractive bias in orientation estimations when previous and current stimuli had similar orientations, and a repulsive bias when they had dissimilar orientations. The attractive bias was markedly reduced-to less than half of its original magnitude-when observers attended to the size, rather than the orientation, of the previous stimulus. Conversely, the repulsive bias for stimuli with large orientation differences was not modulated by feature-based attention. This suggests separate sources of these positive and negative perceptual biases.

Spontaneous repulsive adaptation in the absence of attractive serial dependence.

Abstract: Despite noisy and discontinuous input, vision is remarkably stable and continuous. Recent work suggests that such a remarkable feat is enabled by an active stabilization process integrating information over time, resulting in attractive serial dependence. However, precise mechanisms underlying serial dependence are still unknown. Across three psychophysical experiments, we demonstrate that suppressing high-level modulatory signals on early cortical activity via visual backward masking completely abolishes the serial dependence effect, indicating the critical role of cortical feedback processing on serial dependence. Moreover, we show that the absence of modulatory feedback results in a robust repulsive aftereffect, as in perceptual adaptation, after only 50 ms of stimulation, indicating the presence of a local neurocomputational process for an automatic and spontaneous recalibration of the stimulus representation. These findings collectively illustrate the interplay between two contrasting cortical mechanisms at short timescales that serve as a basis for our perceptual experience.

Temporal attention improves perception similarly at foveal and parafoveal locations.

Abstract: Temporal attention, the prioritization of information at a specific point in time, improves visual performance, but it is unknown whether it does so to the same extent across the visual field. This knowledge is necessary to establish whether temporal attention compensates for heterogeneities in discriminability and speed of processing across the visual field. Discriminability and rate of information accrual depend on eccentricity as well as on polar angle, a characteristic known as performance fields. Spatial attention improves speed of processing more at locations at which discriminability is lower and information accrual is slower, but it improves discriminability to the same extent across isoeccentric locations. Here we asked whether temporal attention benefits discriminability in a similar or differential way across the visual field. Observers were asked to report the orientation of one of two targets presented at different points in time at the same spatial location (fovea, right horizontal meridian, or upper vertical meridian, blocked). Temporal attention improved discriminability and shortened reaction times at the foveal and each parafoveal location similarly. These results provide evidence that temporal attention is similarly effective at multiple locations in the visual field. Consequently, at the tested locations, performance fields are preserved with temporal orienting of attention.

Disambiguating serial effects of multiple timescales.

Abstract: What has been previously experienced can systematically affect human perception in the present. We designed a novel psychophysical experiment to measure the perceptual effects of adapting to dynamically changing stimulus statistics. Observers are presented with a series of oriented Gabor patches and are asked occasionally to judge the orientation of highly ambiguous test patches. We developed a computational model to quantify the influence of past stimuli presentations on the observers' perception of test stimuli over multiple timescales and to show that this influence is distinguishable from simple response biases. The experimental results reveal that perception is attracted toward the very recent past and simultaneously repulsed from stimuli presented at short to medium timescales and attracted to presentations further in the past. All effects differ significantly both on their relative strength and their respective duration. Our model provides a structured way of quantifying serial effects in psychophysical experiments, and it could help experimenters in identifying such effects in their data and distinguish them from less interesting response biases.

Attention explores space periodically at the theta frequency.

Abstract: Voluntary attention is at the core of a wide variety of cognitive functions. Attention can be oriented to and sustained at a location or reoriented in space to allow processing at other locations-critical in an ever-changing environment. Numerous studies have investigated attentional orienting in time and space, but little is known about the spatiotemporal dynamics of attentional reorienting. Here we explicitly manipulated attentional reorienting using a cuing procedure in a two-alternative forced-choice orientation-discrimination task. We interrogated attentional distribution by flashing two probe stimuli with various delays between the precue and target stimuli. Then we used the probabilities that both probes and neither probe were correctly reported to solve a second-degree equation, which estimates the report probability at each probe location. We demonstrated that attention reorients periodically at ∼4 Hz (theta) between the two stimulus locations. We further characterized the processing dynamics at each stimulus location, and demonstrated that attention samples each location periodically at ∼11 Hz (alpha). Finally, simulations support our findings and show that this method is sufficiently powered, making it a valuable tool for studying the spatiotemporal dynamics of attention.

Motion perception as a model for perceptual aging.

Abstract: Research on functional changes across the adult lifespan has been dominated by studies related to cognitive processes. However, it has become evident that a more comprehensive approach to behavioral aging is needed. In particular, our understanding of age-related perceptual changes is limited. Visual motion perception is one of the most studied areas in perceptual aging and therefore, provides an excellent domain on the basis of which we can investigate the complexity of the aging process. We review the existing literature on how aging affects motion perception, including different processing stages, and consider links to cognitive and motor changes. We address the heterogeneity of results and emphasize the role of individual differences. Findings on age-related changes in motion perception ultimately illustrate the complexity of functional dynamics that can contribute to decline as well as stability during healthy aging. We thus propose that motion perception offers a conceptual framework for perceptual aging, encouraging a deliberate consideration of functional limits and resources emerging across the lifespan.

Skeletal representations of shape in human vision: Evidence for a pruned medial axis model.

Abstract: A representation of shape that is low dimensional and stable across minor disruptions is critical for object recognition. Computer vision research suggests that such a representation can be supported by the medial axis-a computational model for extracting a shape's internal skeleton. However, few studies have shown evidence of medial axis processing in humans, and even fewer have examined how the medial axis is extracted in the presence of disruptive contours. Here, we tested whether human skeletal representations of shape reflect the medial axis transform (MAT), a computation sensitive to all available contours, or a pruned medial axis, which ignores contours that may be considered "noise." Across three experiments, participants (N = 2062) were shown complete, perturbed, or illusory two-dimensional shapes on a tablet computer and were asked to tap the shapes anywhere once. When directly compared with another viable model of shape perception (based on principal axes), participants' collective responses were better fit by the medial axis, and a direct test of boundary avoidance suggested that this result was not likely because of a task-specific cognitive strategy (Experiment 1). Moreover, participants' responses reflected a pruned computation in shapes with small or large internal or external perturbations (Experiment 2) and under conditions of illusory contours (Experiment 3). These findings extend previous work by suggesting that humans extract a relatively stable medial axis of shapes. A relatively stable skeletal representation, reflected by a pruned model, may be well equipped to support real-world shape perception and object recognition.

The boundaries of the self: The sense of agency across different sensorimotor aspects.

Abstract: The sense of agency (SoA) is the sensation of control over our actions. SoA is thought to rely mainly upon the comparison of predictions regarding the sensory outcomes of one's actions and the actual sensory outcomes. Previous studies have shown that when a discrepancy is introduced between one's actions and the sensory feedback, the reported SoA is reduced. Experimental manipulations of SoA are typically induced by introducing a discrepancy between a motor action and visual feedback of a specific sensorimotor aspect. For example, introducing a delay or a spatial deviation between the action and its sensory feedback reduces SoA. However, it is yet unclear whether the sensorimotor prediction processes underlying SoA are related between different aspects. Here in one exploratory and one preregistered experiment we tested the sense of agency across temporal, spatial, and anatomical aspects in a within-subject design. Using a novel virtual-reality task allowing the manipulation of the visual feedback of a motor action across different aspects, we show that the sensitivity of agency is different across aspects, agency judgments are correlated across aspects within subjects and bias toward attributing the viewed action to the self or to an external source is correlated as well. Our results suggest that sensorimotor prediction mechanisms underlying SoA are related between different aspects and that people have a predisposition for the directionality of agency judgments. These findings reveal the psychophysical attributes of SoA across sensorimotor aspects. Data and preregistration are available at https://goo.gl/SkbGrb.

Predictive eye movements when hitting a bouncing ball.

Abstract: Predictive eye movements targeted toward the direction of ball bounce are a feature of gaze behavior when intercepting a target soon after it has bounced. However, there is conjecture over the exact location toward which these predictive eye movements are directed, and whether gaze during this period is moving or instead "lies in wait" for the ball to arrive. Therefore, the aim of this study was to further examine the location toward which predictive eye movements are made when hitting a bouncing ball. We tracked the eye and head movements of 23 novice participants who attempted to hit approaching tennis balls in a virtual environment. The balls differed in time from bounce to contact (300, 550, and 800 ms). Results revealed that participants made predictive saccades shortly before the ball bounced in two-thirds of all trials. These saccades were directed several degrees above the position at which the ball bounced, rather than toward the position at which it bounced or toward a position the ball would occupy shortly after the bounce. After the saccade, a separation of roles for the eyes and head ensured that gaze continued to change so that it was as close as possible to the ball soon after bounce. Smooth head movements were responsible for the immediate and ongoing changes in gaze to align it with the ball in the lateral direction, while eye movements realigned gaze with the ball in the vertical direction from approximately 100 ms after the ball changed its direction of motion after bounce. We conclude that predictive saccades direct gaze above the location at which the ball will bounce, presumably in order to facilitate ball tracking after the bounce.

Illumination discrimination for chromatically biased illuminations: Implications for color constancy.

Abstract: We measured discrimination thresholds for illumination changes along different chromatic directions starting from chromatically biased reference illuminations Participants viewed a Mondrian-papered scene illuminated by LED lamps The scene was first illuminated by a reference illumination, followed by two comparisons One comparison matched the reference (the target); the other (the test) varied from the reference, nominally either bluer, yellower, redder, or greener The participant's task was to correctly select the target A staircase procedure found thresholds for discrimination of an illumination change along each axis of chromatic change Nine participants completed the task for five different reference illumination conditions (neutral, blue, yellow, red, and green) We find that relative discrimination thresholds for different chromatic directions of illumination change vary with the reference illumination For the neutral reference, there is a trend for thresholds to be highest in the bluer illumination-change direction, replicating our previous reports of a "blue bias" for neutral reference illuminations For the four chromatic references (blue, yellow, red, and green), the change in illumination toward the neutral reference is less well discriminated than changes in the other directions: a "neutral bias" The results have implications for color constancy: In considering the stability of surface appearance under changes in illumination, both the starting chromaticity of the illumination and direction of change must be considered, as well as the chromatic characteristics of the surface reflectance ensemble They also suggest it will be worthwhile to explore whether and how the human visual system has internalized the statistics of natural illumination changes

The effects of fixational tremor on the retinal image.

Abstract: The study of fixational eye motion has implications for the neural and computational underpinnings of vision. One component of fixational eye motion is tremor, a high-frequency oscillatory jitter reported to be anywhere from ∼11-60 arcseconds in amplitude. In order to isolate the effects of tremor on the retinal image directly and in the absence of optical blur, high-frequency, high-resolution eye traces were collected in six subjects from videos recorded with an adaptive optics scanning laser ophthalmoscope. Videos were acquired while subjects engaged in an active fixation task where they fixated on a tumbling E stimulus and reported changes in its orientation. Spectral analysis was conducted on periods of ocular drift, with all drifts being concatenated together after removal of saccades from the trace. The resultant amplitude spectra showed a slight deviation from the traditional 1/f nature of optical drift in the frequency range of 50-100 Hz, which is indicative of tremor. However, this deviation rarely exceeded 1 arcsecond and the consequent standard deviation of retinal image motion over the tremor band (50-100 Hz) was just over 5 arcseconds. Given such a small amplitude, it is unlikely tremor will contribute in any meaningful way to the visual percept.

Adaptive allocation of human visual working memory capacity during statistical and categorical learning.

Abstract: Human brains are finite, and thus have bounded capacity. An efficient strategy for a capacity-limited agent is to continuously adapt by dynamically reallocating capacity in a task-dependent manner. Here we study this strategy in the context of visual working memory (VWM). People use their VWM stores to remember visual information over seconds or minutes. However, their memory performances are often error-prone, presumably due to VWM capacity limits. We hypothesize that people attempt to be flexible and robust by strategically reallocating their limited VWM capacity based on two factors: (a) the statistical regularities (e.g., stimulus feature means and variances) of the to-be-remembered items, and (b) the requirements of the task that they are attempting to perform. The latter specifies, for example, which types of errors are costly versus irrelevant for task performance. These hypotheses are formalized within a normative computational modeling framework based on rate-distortion theory, an extension of conventional Bayesian approaches that uses information theory to study rate-limited (or capacity-limited) processes. Using images of plants that are naturalistic and precisely controlled, we carried out two sets of experiments. Experiment 1 found that when a stimulus dimension (the widths of plants' leaves) was assigned a distribution, subjects adapted their VWM performances based on this distribution. Experiment 2 found that when one stimulus dimension (e.g., leaf width) was relevant for distinguishing plant categories but another dimension (leaf angle) was irrelevant, subjects' responses in a memory task became relatively more sensitive to the relevant stimulus dimension. Together, these results illustrate the task-dependent robustness of VWM, thereby highlighting the dependence of memory on learning.

The statistics of how natural images drive the responses of neurons.

Abstract: To model the responses of neurons in the early visual system, at least three basic components are required: a receptive field, a normalization term, and a specification of encoding noise. Here, we examine how the receptive field, the normalization factor, and the encoding noise affect the drive to model-neuron responses when stimulated with natural images. We show that when these components are modeled appropriately, the response drives elicited by natural stimuli are Gaussian-distributed and scale invariant, and very nearly maximize the sensitivity (d') for natural-image discrimination. We discuss the statistical models of natural stimuli that can account for these response statistics, and we show how some commonly used modeling practices may distort these results. Finally, we show that normalization can equalize important properties of neural response across different stimulus types. Specifically, narrowband (stimulus- and feature-specific) normalization causes model neurons to yield Gaussian response-drive statistics when stimulated with natural stimuli, 1/f noise stimuli, and white-noise stimuli. The current work makes recommendations for best practices and lays a foundation, grounded in the response statistics to natural stimuli, upon which to build principled models of more complex visual tasks.

Predictive coding of visual motion in both monocular and binocular human visual processing.

Abstract: Neural processing of sensory input in the brain takes time, and for that reason our awareness of visual events lags behind their actual occurrence. One way the brain might compensate to minimize the impact of the resulting delays is through extrapolation. Extrapolation mechanisms have been argued to underlie perceptual illusions in which moving and static stimuli are mislocalised relative to one another (such as the flash-lag and related effects). However, where in the visual hierarchy such extrapolation processes take place remains unknown. Here, we address this question by identifying monocular and binocular contributions to the flash-grab illusion. In this illusion, a brief target is flashed on a moving background that reverses direction. As a result, the perceived position of the target is shifted in the direction of the reversal. We show that the illusion is attenuated, but not eliminated, when the motion reversal and the target are presented dichoptically to separate eyes. This reveals extrapolation mechanisms at both monocular and binocular processing stages contribute to the illusion. We interpret the results in a hierarchical predictive coding framework, and argue that prediction errors in this framework manifest directly as perceptual illusions.

Understanding gloss perception through the lens of art: Combining perception, image analysis, and painting recipes of 17th century painted grapes.

Abstract: To understand the key image features that we use to infer the glossiness of materials, we analyzed the pictorial shortcuts used by 17th century painters to imitate the optical phenomenon of specular reflections when depicting grapes. Gloss perception of painted grapes was determined via a rating experiment. We computed the contrast, blurriness, and coverage of the grapes' highlights in the paintings' images, inspired by Marlow and Anderson (2013). The highlights were manually segmented from the images, and next the features contrast, coverage, and blurriness were semiautomatically quantified using self-defined algorithms. Multiple linear regressions of contrast and blurriness resulted in a predictive model that could explain 69% of the variance in gloss perception. No effect was found for coverage. These findings are in agreement with the instructions to render glossiness of grapes contained in a 17th century painting manual (Beurs, 1692/in press), suggesting that painting practice embeds knowledge about key image features that trigger specific material percepts.

Positive sequential dependency for face attractiveness perception.

Abstract: Recent findings from several groups have demonstrated that visual perception at a given moment can be biased toward what was recently seen. This is true both for basic visual attributes and for more complex representations, such as face identity, gender, or expression. This assimilation to the recent past is a positive serial dependency, similar to a temporal averaging process that capitalizes on short-term correlations in visual input to reduce noise and boost perceptual continuity. Here we examine serial dependencies in face perception using a simple attractiveness rating task and a rapid series of briefly presented face stimuli. In a series of three experiments, our results confirm a previous report that face attractiveness exhibits a positive serial dependency. This intertrial effect is not only determined by face attractiveness on the previous trial, but also depends on the faces shown up to five trials back. We examine the effect of stimulus presentation duration and find that stimuli as brief as 56 ms produce a significant positive dependency similar in magnitude to that produced by stimuli presented for 1,000 ms. We observed stronger positive dependencies between same-gender faces, and found a task dependency: Alternating gender discrimination trials with attractiveness rating trials produced no serial dependency. In sum, these findings show that a perception-stabilizing assimilation effect operates in face attractiveness perception that is task dependent and is acquired surprisingly quickly.

Distinct temporal mechanisms modulate numerosity perception.

Abstract: Our ability to process numerical and temporal information is an evolutionary skill thought to originate from a common magnitude system. In line with a common magnitude system, we have previously shown that adaptation to duration alters numerosity perception. Here, we investigate two hypotheses on how duration influences numerosity perception. A channel-based hypothesis predicts that numerosity perception is influenced by adaptation of onset/offset duration channels which also encode numerosity or wire together with numerosity channels (duration/numerosity channels). Hence, the onset/offset duration of the adapter is driving the effect regardless of the total duration of adaptation. A strength-of-adaptation hypothesis predicts that the effect of duration on numerosity perception is driven by the adaptation of numerosity channels only, with the total duration of adaptation driving the effect regardless of the onset/offset duration of the adapter. We performed two experiments where we manipulated the onset/offset duration of the adapter, the adapter's total presentation time, and the total duration of the adaptation trial. The first experiment tested the effect of adaptation to duration on numerosity discrimination, whereas the second experiment tested the effect of adaptation to numerosity and duration on numerosity discrimination. We found that the effect of adaptation to duration on numerosity perception is primarily driven by adapting duration/numerosity channels, supporting the channel-based hypothesis. In contrast, the effect of adaptation to numerosity on numerosity perception appears to be driven by the total duration of the adaptation trial, supporting the strength-of-adaptation hypothesis. Thus, we show that adaptation of at least two temporal mechanisms influences numerosity perception.

Decoding go/no-go decisions from eye movements.

Abstract: Neural activity in brain areas involved in the planning and execution of eye movements predicts the outcome of an upcoming perceptual decision. Many real-world decisions, such as whether to swing at a baseball pitch, are accompanied by characteristic eye-movement behavior. Here we ask whether human eye-movement kinematics can sensitively predict decision outcomes in a go/no-go task requiring rapid interceptive hand movements. Observers (n = 45) viewed a moving target that passed through or missed a designated strike box. Critically, the target disappeared briefly after launch, and observers had to predict the target's trajectory, withholding a hand movement if it missed (no-go) or intercepting inside the strike box (go). We found that go/no-go decisions were reflected in distinct eye-movement responses on a trial-by-trial basis: Eye-position error and targeting-saccade dynamics predicted decision outcome with 76% accuracy across conditions. Model prediction accuracy was related to observers' decision accuracy across different levels of task difficulty and sensory-signal strength. Our findings suggest that eye movements provide a sensitive and continuous readout of internal neural decision-making processes and reflect decision-task requirements in human observers.

A systematic approach to testing and predicting light-material interactions

Abstract: Photographers and lighting designers set up lighting environments that best depict objects and human figures to convey key aspects of the visual appearance of various materials, following rules drawn from experience. Understanding which lighting environment is best adapted to convey which key aspects of materials is an important question in the field of human vision. The endless range of natural materials and lighting environments poses a major problem in this respect. Here we present a systematic approach to make this problem tractable for lighting-material interactions, using optics-based models composed of canonical lighting and material modes. In two psychophysical experiments, different groups of inexperienced observers judged the material qualities of the objects depicted in the stimulus images. In the first experiment, we took photographs of real objects as stimuli under canonical lightings. In a second experiment, we selected three generic natural lighting environments on the basis of their predicted lighting effects and made computer renderings of the objects. The selected natural lighting environments have characteristics similar to the canonical lightings, as computed using a spherical harmonic analysis. Results from the two experiments correlate strongly, showing (a) how canonical material and lighting modes associate with perceived material qualities; and (b) which lighting is best adapted to evoke perceived material qualities, such as softness, smoothness, and glossiness. Our results demonstrate that a system of canonical modes spanning the natural range of lighting and materials provides a good basis to study lighting-material interactions in their full natural ecology.

Category selectivity for animals and man-made objects: Beyond low- and mid-level visual features.

Abstract: Distinct concepts, such as animate and inanimate entities, comprise vast and systematic differences in visual features. While observers search for animals faster than man-made objects, it remains unclear to what extent visual or conceptual information contributes to such differences in visual search performance. Previous studies demonstrated that visual features are likely sufficient for distinguishing animals from man-made objects. Across four experiments, we examined whether low- or mid-level visual features solely contribute to the search advantage for animals by using images of comparable visual shape and gist statistics across the categories. Participants searched for either animal or man-made object targets on a multiple-item display with fruit/vegetable distractors. We consistently observed faster search performance for animal than man-made object targets. Such advantage for animals was unlikely affected by differences in low- or mid-level visual properties or whether observers were either explicitly told about the specific targets or not explicitly told to search for either animals or man-made objects. Instead, the efficiency in categorizing animals over man-made objects appeared to contribute to the search advantage. We suggest that apart from low- or mid-level visual differences among categories, higher-order processes, such as categorization via interpreting visual inputs and mapping them onto distinct concepts, may be critical in shaping category-selective effects.