Successful speech perception requires that listeners map the acoustic signal to linguistic categories. These mappings are not only probabilistic, but change depending on the situation. For example, one talker's /p/ might be physically indistinguishable from another talker's /b/ (cf. lack of invariance). We characterize the computational problem posed by such a subjectively nonstationary world and propose that the speech perception system overcomes this challenge by (a) recognizing previously encountered situations, (b) generalizing to other situations based on previous similar experience, and (c) adapting to novel situations. We formalize this proposal in the ideal adapter framework: (a) to (c) can be understood as inference under uncertainty about the appropriate generative model for the current talker, thereby facilitating robust speech perception despite the lack of invariance. We focus on 2 critical aspects of the ideal adapter. First, in situations that clearly deviate from previous experience, listeners need to adapt. We develop a distributional (belief-updating) learning model of incremental adaptation. The model provides a good fit against known and novel phonetic adaptation data, including perceptual recalibration and selective adaptation. Second, robust speech recognition requires that listeners learn to represent the structured component of cross-situation variability in the speech signal. We discuss how these 2 aspects of the ideal adapter provide a unifying explanation for adaptation, talker-specificity, and generalization across talkers and groups of talkers (e.g., accents and dialects). The ideal adapter provides a guiding framework for future investigations into speech perception and adaptation, and more broadly language comprehension.

Robust speech perception: recognize the familiar, generalize to the similar, and adapt to the novel.

The ability to group items and events into functional categories is a fundamental characteristic of sophisticated thought. It is subserved by plasticity in many neural systems, including neocortical regions (sensory, prefrontal, parietal, and motor cortex), the medial temporal lobe, the basal ganglia, and midbrain dopaminergic systems. These systems interact during category learning. Corticostriatal loops may mediate recursive, bootstrapping interactions between fast reward-gated plasticity in the basal ganglia and slow reward-shaded plasticity in the cortex. This can provide a balance between acquisition of details of experiences and generalization across them. Interactions between the corticostriatal loops can integrate perceptual, response, and feedback-related aspects of the task and mediate the shift from novice to skilled performance. The basal ganglia and medial temporal lobe interact competitively or cooperatively, depending on the demands of the learning task.

/pdf/category-learning-in-the-brain-58925cumz0.pdf

Category Learning in the Brain

https://www.cell.com/neuron/pdf/S0896-6273(10)00170-4.pdf

The developmental origins of voice processing in the human brain

The development of speech perception shows a dramatic transition between infancy and adulthood. Between 6 and 12 months, infants' initial ability to discriminate all phonetic units across the world's languages narrows—native discrimination increases while non-native discrimination shows a steep decline. We used magnetoencephalography (MEG) to examine whether brain oscillations in the theta band (4–8 Hz), reflecting increases in attention and cognitive effort, would provide a neural measure of the perceptual narrowing phenomenon in speech. Using an oddball paradigm, we varied speech stimuli in two dimensions, stimulus frequency (frequent vs. infrequent) and language (native vs. non-native speech syllables) and tested 6-month-old infants, 12-month-old infants, and adults. We hypothesized that 6-month-old infants would show increased relative theta power (RTP) for frequent syllables, regardless of their status as native or non-native syllables, reflecting young infants' attention and cognitive effort in response to highly frequent stimuli (“statistical learning”). In adults, we hypothesized increased RTP for non-native stimuli, regardless of their presentation frequency, reflecting increased cognitive effort for non-native phonetic categories. The 12-month-old infants were expected to show a pattern in transition, but one more similar to adults than to 6-month-old infants. The MEG brain rhythm results supported these hypotheses. We suggest that perceptual narrowing in speech perception is governed by an implicit learning process. This learning process involves an implicit shift in attention from frequent events (infants) to learned categories (adults). Theta brain oscillatory activity may provide an index of perceptual narrowing beyond speech, and would offer a test of whether the early speech learning process is governed by domain-general or domain-specific processes.

Theta brain rhythms index perceptual narrowing in infant speech

Most theories of categorization emphasize how continuous perceptual information is mapped to categories. However, equally important are the informational assumptions of a model, the type of information subserving this mapping. This is crucial in speech perception where the signal is variable and context dependent. This study assessed the informational assumptions of several models of speech categorization, in particular, the number of cues that are the basis of categorization and whether these cues represent the input veridically or have undergone compensation. We collected a corpus of 2,880 fricative productions (Jongman, Wayland, & Wong, 2000) spanning many talker and vowel contexts and measured 24 cues for each. A subset was also presented to listeners in an 8AFC phoneme categorization task. We then trained a common classification model based on logistic regression to categorize the fricative from the cue values and manipulated the information in the training set to contrast (a) models based on a small number of invariant cues, (b) models using all cues without compensation, and (c) models in which cues underwent compensation for contextual factors. Compensation was modeled by computing cues relative to expectations (C-CuRE), a new approach to compensation that preserves fine-grained detail in the signal. Only the compensation model achieved a similar accuracy to listeners and showed the same effects of context. Thus, even simple categorization metrics can overcome the variability in speech when sufficient information is available and compensation schemes like C-CuRE are employed.

https://kuscholarworks.ku.edu/bitstream/handle/1808/13410/McMuray_Jongman_2011.pdf;sequence=1

What information is necessary for speech categorization? Harnessing variability in the speech signal by integrating cues computed relative to expectations

Infants rapidly learn the sound categories of their native language, even though they do not receive explicit or focused training. Recent research suggests that this learning is due to infants' sensitivity to the distribution of speech sounds and that infant-directed speech contains the distributional information needed to form native-language vowel categories. An algorithm, based on Expectation–Maximization, is presented here for learning the categories from a sequence of vowel tokens without (i) receiving any category information with each vowel token, (ii) knowing in advance the number of categories to learn, or (iii) having access to the entire data ensemble. When exposed to vowel tokens drawn from either English or Japanese infant-directed speech, the algorithm successfully discovered the language-specific vowel categories (/i, i, e, e/ for English, /i, iː, e, eː/ for Japanese). A nonparametric version of the algorithm, closely related to neural network models based on topographic representation and competitive Hebbian learning, also was able to discover the vowel categories, albeit somewhat less reliably. These results reinforce the proposal that native-language speech categories are acquired through distributional learning and that such learning may be instantiated in a biologically plausible manner. 

language acquisition
speech perception
expectation maximization
online learning

/pdf/unsupervised-learning-of-vowel-categories-from-infant-qg9mw491x2.pdf

Unsupervised learning of vowel categories from infant-directed speech

These experiments investigate whether the perceptual boundary between [a] and [a:] is affected by the differing phonotactics of the Sino-Japanese and Foreign strata of the Japanese lexicon. We presented a range of edited natural vowels from [a] to [a:] at the end of carrier nonwords of the form [CoC'__] and asked subjects to judge whether the final vowel was long or short, while C and C' were varied to make 9 carriers ranging from very Sino-Japanese-like to very Foreign-like. The perceptual boundary between [a] and [a:] was affected by both C and C', shifting through about 20 msec—a larger and more robust effect than was obtained in a word-superiority experiment with the same paradigm and subjects. These results cast doubt on explanations of phonotactic effects based on lexical-activation spreading. The influence of C at a distance of three phonemes from the ambiguous segment cannot be explained by simple segment-to-segment transitional probabilities.

Phonotactics in the perception of Japanese vowel length: evidence for long-distance dependencies.

A familiarity database was developed for a bout 80,000 Japanese words of which familiarity scores were rated by 32 Japanese adults using a 7-point scale in auditory, visual, and audio-visual modalities. Auditory, visual, and audio-visual stimulus words were selected from the database according to their word familiarity for size estimation of the mental lexicon. Sixty Japanese adults participated in a two-alternative forcedchoice task (Know–Don't know) for the stimulus words. The size of the mental lexicon was estimated as the number of words of which familiarity is a bove a particular word corresponding to 50% point on the f itted logistic curve to “know”-response probab ility of the stimulus words. The estimated size was about 68,000 for auditory words, and about 66,000 both for visual and audio-visual words when homophones and homographs were included. The results suggest that very small difference in the mental lexicon size among moda lities.

Estimation of mental lexicon size with word familiarity database

The theory of relational acoustic invariance [Pickett, E. R., et al. (1999). Phonetica 56, 135-157] was tested with the Japanese stop quantity distinction in disyllables spoken at various rates. The questions were whether the perceptual boundary between the two phonemic categories of single and geminate stops is invariant across rates, and whether there is a close correspondence between the perception and production boundaries. The durational ratio of stop closure to word (where the "word" was defined as disyllables) was previously found to be an invariant parameter that classified the two categories in production, but the present study found that this ratio varied with different speaking rates in perception. However, regression and discriminant analyses of perception and production data showed that treating stop closure as a function of word duration with an intercept term represented the perception and production boundaries very well. This result indicated that the durational ratio of adjusted stop closure (i.e., closure with an added constant) to the word was invariant and distinguished the two phonemic categories clearly. Taken together, the results support the relational acoustic invariance theory, and help refine the theory with regard to exactly what form 'invariance' can take.

Shigeaki Amano

Papers

Unsupervised learning of vowel categories from infant-directed speech

Phonotactics in the perception of Japanese vowel length: evidence for long-distance dependencies.

Estimation of mental lexicon size with word familiarity database

Perception and production boundaries between single and geminate stops in Japanese

Relationships between Depression and Stress Factors in Housework and Paid Work among Japanese Women.