Generative adversarial networks (GANs) are a hot research topic recently. GANs have been widely studied since 2014, and a large number of algorithms have been proposed. However, there is few comprehensive study explaining the connections among different GANs variants, and how they have evolved. In this paper, we attempt to provide a review on various GANs methods from the perspectives of algorithms, theory, and applications. Firstly, the motivations, mathematical representations, and structure of most GANs algorithms are introduced in details. Furthermore, GANs have been combined with other machine learning algorithms for specific applications, such as semi-supervised learning, transfer learning, and reinforcement learning. This paper compares the commonalities and differences of these GANs methods. Secondly, theoretical issues related to GANs are investigated. Thirdly, typical applications of GANs in image processing and computer vision, natural language processing, music, speech and audio, medical field, and data science are illustrated. Finally, the future open research problems for GANs are pointed out.

A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications

The inferotemporal (IT) cortex is responsible for object recognition, but it is unclear how the representation of visual objects is organized in this part of the brain. Areas that are selective for categories such as faces, bodies, and scenes have been found1-5, but large parts of IT cortex lack any known specialization, raising the question of what general principle governs IT organization. Here we used functional MRI, microstimulation, electrophysiology, and deep networks to investigate the organization of macaque IT cortex. We built a low-dimensional object space to describe general objects using a feedforward deep neural network trained on object classification6. Responses of IT cells to a large set of objects revealed that single IT cells project incoming objects onto specific axes of this space. Anatomically, cells were clustered into four networks according to the first two components of their preferred axes, forming a map of object space. This map was repeated across three hierarchical stages of increasing view invariance, and cells that comprised these maps collectively harboured sufficient coding capacity to approximately reconstruct objects. These results provide a unified picture of IT organization in which category-selective regions are part of a coarse map of object space whose dimensions can be extracted from a deep network.

/pdf/a-map-of-object-space-in-primate-inferotemporal-cortex-50plzt6j1p.pdf

A map of object space in primate inferotemporal cortex.

Here we report that monkeys raised without exposure to faces did not develop face domains, but did develop domains for other categories and did show normal retinotopic organization, indicating that early face deprivation leads to a highly selective cortical processing deficit. Therefore, experience must be necessary for the formation (or maintenance) of face domains. Gaze tracking revealed that control monkeys looked preferentially at faces, even at ages prior to the emergence of face domains, but face-deprived monkeys did not, indicating that face looking is not innate. A retinotopic organization is present throughout the visual system at birth, so selective early viewing behavior could bias category-specific visual responses toward particular retinotopic representations, thereby leading to domain formation in stereotyped locations in inferotemporal cortex, without requiring category-specific templates or biases. Thus, we propose that environmental importance influences viewing behavior, viewing behavior drives neuronal activity, and neuronal activity sculpts domain formation.

Seeing faces is necessary for face-domain formation.

Human object-selective cortex shows a large-scale organization characterized by the high-level properties of both animacy and object size. To what extent are these neural responses explained by primitive perceptual features that distinguish animals from objects and big objects from small objects? To address this question, we used a texture synthesis algorithm to create a class of stimuli-texforms-which preserve some mid-level texture and form information from objects while rendering them unrecognizable. We found that unrecognizable texforms were sufficient to elicit the large-scale organizations of object-selective cortex along the entire ventral pathway. Further, the structure in the neural patterns elicited by texforms was well predicted by curvature features and by intermediate layers of a deep convolutional neural network, supporting the mid-level nature of the representations. These results provide clear evidence that a substantial portion of ventral stream organization can be accounted for by coarse texture and form information without requiring explicit recognition of intact objects.

https://www.pnas.org/content/pnas/115/38/E9015.full.pdf

Mid-level visual features underlie the high-level categorical organization of the ventral stream.

Artificial vision has often been described as one of the key remaining challenges to be solved before machines can act intelligently. Recent developments in a branch of machine learning known as deep learning have catalyzed impressive gains in machine vision-giving a sense that the problem of vision is getting closer to being solved. The goal of this review is to provide a comprehensive overview of recent deep learning developments and to critically assess actual progress toward achieving human-level visual intelligence. I discuss the implications of the successes and limitations of modern machine vision algorithms for biological vision and the prospect for neuroscience to inform the design of future artificial vision systems.

Deep Learning: The Good, the Bad, and the Ugly.

Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences.

Neurons in primate inferotemporal cortex (IT) are clustered into patches of shared image preferences. Functional imaging has shown that these patches are activated by natural categories (e.g., faces, body parts, and places), artificial categories (numerals, words) and geometric features (curvature and real-world size). These domains develop in the same cortical locations across monkeys and humans, which raises the possibility of common innate mechanisms. Although these commonalities could be high-level template-based categories, it is alternatively possible that the domain locations are constrained by low-level properties such as end-stopping, eccentricity, and the shape of the preferred images. To explore this, we looked for correlations among curvature preference, receptive field (RF) end-stopping, and RF eccentricity in the ventral stream. We recorded from sites in V1, V4, and posterior IT (PIT) from six monkeys using microelectrode arrays. Across all visual areas, we found a tendency for end-stopped sites to prefer curved over straight contours. Further, we found a progression in population curvature preferences along the visual hierarchy, where, on average, V1 sites preferred straight Gabors, V4 sites preferred curved stimuli, and many PIT sites showed a preference for curvature that was concave relative to fixation. Our results provide evidence that high-level functional domains may be mapped according to early rudimentary properties of the visual system. SIGNIFICANCE STATEMENT The macaque occipitotemporal cortex contains clusters of neurons with preferences for categories such as faces, body parts, and places. One common question is how these clusters (or “domains”) acquire their cortical position along the ventral stream. We and other investigators previously established an fMRI-level correlation among these category domains, retinotopy, and curvature preferences: for example, in inferotemporal cortex, face- and curvature-preferring domains show a central visual field bias whereas place- and rectilinear-preferring domains show a more peripheral visual field bias. Here, we have found an electrophysiological-level explanation for the correlation among domain preference, curvature, and retinotopy based on neuronal preference for short over long contours, also called end-stopping.

https://www.jneurosci.org/content/jneuro/37/3/648.full.pdf

End-Stopping Predicts Curvature Tuning along the Ventral Stream.

Perceptual stability requires the integration of information across eye movements We first tested the hypothesis that motion signals are integrated by neurons whose receptive fields (RFs) do not move with the eye but stay fixed in the world Specifically, we measured the RF properties of neurons in the middle temporal area (MT) of macaques (Macaca mulatta) during the slow phase of optokinetic nystagmus Using a novel method to estimate RF locations for both spikes and local field potentials, we found that the location on the retina that changed spike rates or local field potentials did not change with eye position; RFs moved with the eye Second, we tested the hypothesis that neurons link information across eye positions by remapping the retinal location of their RFs to future locations To test this, we compared RF locations during leftward and rightward slow phases of optokinetic nystagmus We found no evidence for remapping during slow eye movements; the RF location was not affected by eye-movement direction Together, our results show that RFs of MT neurons and the aggregate activity reflected in local field potentials are yoked to the eye during slow eye movements This implies that individual MT neurons do not integrate sensory information from a single position in the world across eye movements Future research will have to determine whether such integration, and the construction of perceptual stability, takes place in the form of a distributed population code in eye-centered visual cortex or is deferred to downstream areas

https://www.jneurosci.org/content/jneuro/31/29/10437.full.pdf

Receptive field positions in area MT during slow eye movements.

How we perceive the world as stable despite the frequent disruptions of the retinal image caused by eye movements is one of the fundamental questions in sensory neuroscience. Seemingly convergent evidence points towards a mechanism which dynamically updates representations of visual space in anticipation of a movement (Wurtz, 2008). In particular, receptive fields (RFs) of neurons, predominantly within oculomotor and attention related brain structures (Duhamel et al., 1992; Walker et al., 1995; Umeno & Goldberg, 1997), are thought to ‘remap’ to their future, post-movement location prior to an impending eye movement. New studies (Neupane et al., 2016a,b) report observations on RF dynamics at the time of eye movements of neurons in area V4. These dynamics are interpreted as being largely dominated by a remapping of RFs. Critically, these observations appear at odds with a previous study reporting a different type of RF dynamics within the same brain structure (Tolias et al., 2001), consisting of a shrinkage and shift of RFs towards the movement target. Importantly, RFs have been measured with different techniques in those studies. Here, we measured V4 RFs comparable to Neupane et al. (2016a,b) and observe a shrinkage and shift of RFs towards the saccade target when analyzing the immediate stimulus response (Zirnsak et al., 2014). When analyzing the late stimulus response (Neupane et al., 2016a,b), we observe RF shifts resembling remapping. We discuss possible causes for these shifts and point out important issues which future studies on RF dynamics need to address.

/pdf/two-types-of-receptive-field-dynamics-in-area-v4-at-the-time-36wmqgbexe.pdf

Two Types of Receptive Field Dynamics in Area V4 at the Time of Eye Movements

http://www.hms.harvard.edu/bss/neuro/bornlab/lab/papers/born-trott-hartmann-cortical_visual_prosthesis-visionres2014.pdf

Till S. Hartmann

Papers

Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences.

End-Stopping Predicts Curvature Tuning along the Ventral Stream.

Receptive field positions in area MT during slow eye movements.

Two Types of Receptive Field Dynamics in Area V4 at the Time of Eye Movements

Cortical magnification plus cortical plasticity equals vision