The organization of behavior: A neuropsychological theory

Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

https://www.robots.ox.ac.uk/~vgg/data/dtd/doc/textures-poster-cvpr14.pdf

Describing Textures in the Wild

We study the problem of object recognition for categories for which we have no training examples, a task also called zero--data or zero-shot learning. This situation has hardly been studied in computer vision research, even though it occurs frequently; the world contains tens of thousands of different object classes, and image collections have been formed and suitably annotated for only a few of them. To tackle the problem, we introduce attribute-based classification: Objects are identified based on a high-level description that is phrased in terms of semantic attributes, such as the object's color or shape. Because the identification of each such property transcends the specific learning task at hand, the attribute classifiers can be prelearned independently, for example, from existing image data sets unrelated to the current task. Afterward, new classes can be detected based on their attribute representation, without the need for a new training phase. In this paper, we also introduce a new data set, Animals with Attributes, of over 30,000 images of 50 animal classes, annotated with 85 semantic attributes. Extensive experiments on this and two more data sets show that attribute-based classification indeed is able to categorize images without access to any training images of the target classes.

Attribute-Based Classification for Zero-Shot Visual Object Categorization

Natural Neural Projection Dynamics Underlying Social Behavior

Human-nameable visual “attributes” can benefit various recognition tasks. However, existing techniques restrict these properties to categorical labels (for example, a person is ‘smiling’ or not, a scene is ‘dry’ or not), and thus fail to capture more general semantic relationships. We propose to model relative attributes. Given training data stating how object/scene categories relate according to different attributes, we learn a ranking function per attribute. The learned ranking functions predict the relative strength of each property in novel images. We then build a generative model over the joint space of attribute ranking outputs, and propose a novel form of zero-shot learning in which the supervisor relates the unseen object category to previously seen objects via attributes (for example, ‘bears are furrier than giraffes’). We further show how the proposed relative attributes enable richer textual descriptions for new images, which in practice are more precise for human interpretation. We demonstrate the approach on datasets of faces and natural scenes, and show its clear advantages over traditional binary attribute prediction for these new tasks.

Relative attributes

It is common to use domain specific terminology - attributes - to describe the visual appearance of objects. In order to scale the use of these describable visual attributes to a large number of categories, especially those not well studied by psychologists or linguists, it will be necessary to find alternative techniques for identifying attribute vocabularies and for learning to recognize attributes without hand labeled training data. We demonstrate that it is possible to accomplish both these tasks automatically by mining text and image data sampled from the Internet. The proposed approach also characterizes attributes according to their visual representation: global or local, and type: color, texture, or shape. This work focuses on discovering attributes and their visual appearance, and is as agnostic as possible about the textual description.

/pdf/automatic-attribute-discovery-and-characterization-from-3jeing4ixc.pdf

Automatic attribute discovery and characterization from noisy web data

We analyzed the receptive field information conveyed by interspike intervals (ISIs) in the auditory cortex. In the visual system, different ISIs may both code for different visual features and convey differing amounts of stimulus information. To determine their potential role in auditory signal processing, we obtained extracellular recordings in the primary auditory cortex (AI) of the cat while presenting a dynamic moving ripple stimulus and then used the responses to construct spectrotemporal receptive fields (STRFs). For each neuron, we constructed three STRFs, one for short-ISI events (ISI 15 ms); and one including all events. To characterize stimulus encoding, we calculated the feature selectivity and event information for each of the STRFs. Short-ISI spikes were more feature selective and conveyed information more efficiently. The different ISI regimens of AI neurons did not represent different stimulus features, but short-ISI spike events did contribute over-proportionately to the full spike train STRF information. Thus short-ISIs constitute a robust representation of auditory features, and they are particularly effective at driving postsynaptic activity. This suggests that short-ISI events are especially suited to provide noise immunity and high-fidelity information transmission in AI.

https://www.physiology.org/doi/pdf/10.1152/jn.01055.2010

Improved stimulus representation by short interspike intervals in primary auditory cortex.

Cochlear implant electrical stimulation of the auditory system to rehabilitate deafness has been remarkably successful. Its deployment requires both an intact auditory nerve and a suitably patent cochlear lumen. When disease renders prerequisite conditions impassable, such as in neurofibromatosis type II and cochlear obliterans, alternative treatment targets are considered. Electrical stimulation of the cochlear nucleus and midbrain in humans has delivered encouraging clinical outcomes, buttressing the promise of central auditory prostheses to mitigate deafness in those who are not candidates for cochlear implantation. In this study we explored another possible implant target: the auditory thalamus. In anesthetized cats, we first presented pure tones to determine frequency preferences of thalamic and cortical sites. We then electrically stimulated tonotopically organized thalamic sites while recording from primary auditory cortical sites using a multichannel recording probe. Cathode-leading biphasic thalamic stimulation thresholds that evoked cortical responses were much lower than published accounts of cochlear and midbrain stimulation. Cortical activation dynamic ranges were similar to those reported for cochlear stimulation, but they were narrower than those found through midbrain stimulation. Our results imply that thalamic stimulation can activate auditory cortex at low electrical current levels and suggest an auditory thalamic implant may be a viable central auditory prosthesis.

/pdf/primary-auditory-cortical-responses-to-electrical-1hgz7swc6d.pdf

Primary auditory cortical responses to electrical stimulation of the thalamus.

Inhibitory interneurons constitute ∼20% of auditory cortical cells and are essential for shaping sensory processing. Connectivity patterns of interneurons in relation to functional organization principles are not well understood. We contrasted the connection patterns of parvalbumin-immunoreactive cells in two functionally distinct cortical regions: the tonotopic, narrowly frequency-tuned module [central narrow band (cNB)] of cat central primary auditory cortex (AI) and the nontonotopic, broadly tuned second auditory field (AII). Interneuronal connectivity patterns and laminar distribution were identified by combining a retrograde tracer (wheat-germ agglutinin apo-horseradish peroxidase colloidal gold) with labeling of the Ca(2+) binding protein parvalbumin (Pv), a marker for the GABAergic interneurons usually described physiologically as fast-spiking neurons. In AI, parvalbumin-positive (Pv+) cells constituted 13% of the retrograde labeled cells in the immediate vicinity of the injection site, compared to 10% in AII. The retrograde labeling of Pv+ cells along isofrequency countours was confined to the cNB. The spatial spread of labeled excitatory neurons in AI was more than twice that found for Pv+ cells. By contrast, in the AII, the spread of Pv+ cells was nearly equal to that of excitatory neurons. The retrograde labeling of Pv+ cells was anisotropic in AI and isotropic in AII. This demonstration of inhibitory networks in auditory cortex reveals that the connections of cat GABAergic AI and AII cells follow different anatomical plans and thus contribute differently to the shaping of neural response properties. The finding that local connectivity of parvalbumin-immunoreactive neurons in AI is closely aligned with spectral integration properties demonstrates the critical role of inhibition in creating distinct processing modules in AI.

https://www.jneurosci.org/content/jneuro/31/37/13333.full.pdf

Functional Networks of Parvalbumin-Immunoreactive Neurons in Cat Auditory Cortex

Studies of patterned spontaneous activity can elucidate how the organization of neural circuits emerges. Using in vivo two-photon Ca2+ imaging, we studied spatio-temporal patterns of spontaneous activity in the optic tectum of Xenopus tadpoles. We found rhythmic patterns of global synchronous spontaneous activity between neurons, which depends on visual experience and developmental stage. By contrast, synchronous spontaneous activity between non-neuronal cells is mediated more locally. To understand the source of the neuronal spontaneous activity, input to the tectum was systematically removed. Whereas removing input from the visual or mechanosensory system alone had little effect on patterned spontaneous activity, removing input from both systems drastically altered it. These results suggest that either input is sufficient to maintain the intrinsically generated spontaneous activity and that patterned spontaneous activity results from input from multisensory systems. Thus, the amphibian midbrain differs from the mammalian visual system, whose spontaneous activity is controlled by retinal waves.

/pdf/global-hyper-synchronous-spontaneous-activity-in-the-6784c6xty3.pdf

Jonathan Y. Shih

Papers

Automatic attribute discovery and characterization from noisy web data

Improved stimulus representation by short interspike intervals in primary auditory cortex.

Primary auditory cortical responses to electrical stimulation of the thalamus.

Functional Networks of Parvalbumin-Immunoreactive Neurons in Cat Auditory Cortex

Global Hyper-synchronous Spontaneous Activity in the Developing Optic Tectum