Understanding human individuation of unfamiliar faces with oddball fast periodic visual stimulation and electroencephalography

TLDR: Overall, the oddball FPVS paradigm is recommended as an alternative approach to behavioral or traditional event‐related potential EEG measures of face individuation, including its (a)typical development, with early studies supporting the application of this technique to clinical testing (e.g., autism spectrum disorder).

Abstract: To investigate face individuation (FI), a critical brain function in the human species, an oddball fast periodic visual stimulation (FPVS) approach was recently introduced (Liu-Shuang et al., Neuropsychologia, 2014, 52, 57). In this paradigm, an image of an unfamiliar "base" facial identity is repeated at a rapid rate F (e.g., 6 Hz) and different unfamiliar "oddball" facial identities are inserted every nth item, at a F/n rate (e.g., every 5th item, 1.2 Hz). This stimulation elicits FI responses at F/n and its harmonics (2F/n, 3F/n, etc.), reflecting neural discrimination between oddball versus base facial identities, which is quantified in the frequency domain of the electroencephalogram (EEG). This paradigm, used in 20 published studies, demonstrates substantial advantages for measuring FI in terms of validity, objectivity, reliability, and sensitivity. Human intracerebral recordings suggest that this FI response originates from neural populations in the lateral inferior occipital and fusiform gyri, with a right hemispheric dominance consistent with the localization of brain lesions specifically affecting facial identity recognition (prosopagnosia). Here, we summarize the contributions of the oddball FPVS framework toward understanding FI, including its (a)typical development, with early studies supporting the application of this technique to clinical testing (e.g., autism spectrum disorder). This review also includes an in-depth analysis of the paradigm's methodology, with guidelines for designing future studies. A large-scale group analysis compiling data across 130 observers provides insights into the oddball FPVS FI response properties. Overall, we recommend the oddball FPVS paradigm as an alternative approach to behavioral or traditional event-related potential EEG measures of face individuation.

Figures

Figure 11. Face individuation response localization predictive of transient palinopsia provoked by electrical stimulation during intracerebral recording (from Jonas et al., 2018). A. Localization of intracerebral electrode contacts implanted in a patient with pharmaco-resistant epilepsy. The localization of the electrode F is shown, relative to face-selective regions, defined with an fMRI functional localizer in this patient. The contacts F1-F2 fall within the right FFA. B. The patient completed 4 sequences of the FI paradigm reviewed here. Contact F1 showed the strongest FI response, followed by F2, as demonstrated by the clear peaks at the oddball face identity change frequency and its harmonics in the frequency-domain SNR spectra. C. Electrical stimulation of contacts F1-F2 only elicited vivid face hallucinations in the patient, who reported seeing familiar facial identities (that she was unable to identify) mixed with currently viewed faces (palinopsia) (see Jonas et al., 2018 for methodology).

Figure 2. Example of a set of homogenous individual female faces used in face individuation paradigms (from Retter et al., in preparation).

Figure 21. General data analysis procedure. Following (1) data acquisition, the raw EEG data is usually (2) preprocessed (i.e., low/high pass filter; re-reference; blink removal with independent component analysis …) and segmented to contain only the relevant stimulation duration with an integer number of oddball frequency cycles. These epochs are then averaged by condition/subject and (3) transformed into the frequency-domain with FFT. From the resulting amplitude spectrum, (4) “chunks” containing the relevant

Figure 18. Typical and atypical development of FI (from Lochy et al., 2020). A. Typical development. The amplitude spectrum shows responses obtained in 5-year-old children (preschoolers) in the oddball FPVS paradigm. Large peaks at F and its harmonics are found for the general visual response (6 Hz, 12 Hz, etc.). However, the F/n FI response is predominantly distributed only over the first harmonic (1.2 Hz), contrarily to adults (Figure 4). Additionally, the children’s FI response is much less reduced with face inversion (11%) compared to older children (8-12 years old, Figure 18B) and adults (Figure 7). B. Atypical development (from Vettori et al., 2019b). The oddball FPVS paradigm reveals multiple dissociations in the FI response from 8- to 12-year-old boys that are either typically developing (TD) or with autism spectrum disorder (ASD). The FI response for upright faces is significantly larger in the TD than the ASD group (see the subtraction: TD – ASD), but the groups do not differ in response to inverted faces. Moreover, only the TD group show a significant face inversion effect. Data from the frequency spectra are taken from a bilateral occipito-temporal ROI shown on the 3D scalp topographies on the right panel.

Figure 16. Composite effect (from Rossion et al., in preparation). A. Example of composite face stimuli. The top half of one facial identity is paired with bottom halves from different unfamiliar facial identities, with a thin white space separating both halves (see Rossion & Retter, 2015; these stimuli are available at: https://face-categorization-lab.webnode.com/resources/). In total there

Figure 8. Generalization of the face individuation response across changes in head views (from Or et al., in preparation). A. Stimuli were a set of 11 female facial identities from the MPI database of 3D scanned faces. For each face, 37 head views were used: -90° (left profile) to 90° (right profile) in steps of 5°. In the experimental paradigm, face stimuli were shown at a rapid 6 Hz rate (i.e., 6 images/s) through sinusoidal contrast modulation, and varied randomly in size at each presentation. A randomly-chosen base facial identity (A) was repeated throughout the sequence, with different facial identities (C, F…) inserted as every 7th face (0.86 Hz). Example sequences from three view conditions (0°, ±45°, ±90°) are shown here (7 conditions in the actual experiment: 0°, ±15°, ±30°, ±45°, ±60°, ±75°, ±95°). In the 0° condition, all face stimuli were shown at a full-front view without head view change. In the ±45° and ±90° conditions, each face stimulus varied randomly within the range of -45° to +45°, and -90° and +90°, respectively. Periodic EEG responses at the stimulus presentation frequency (F = 6 Hz) and its harmonics reflect responses to the onset and offset of face stimuli, whereas responses at the face identity change frequency (F/n = 0.86 Hz) and harmonics specifically reflect FI across head view variations. B. The effect of head view on the FI response. Top panel: Observer-averaged scalp topographies showing response amplitude reduction with increasing view

Full text

HAL Id: hal-02931197
https://hal.archives-ouvertes.fr/hal-02931197
Submitted on 24 Nov 2020
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Understanding human individuation of unfamiliar faces
with oddball fast periodic visual stimulation and
electroencephalography
Bruno Rossion, Talia Retter, Joan Liu-shuang
To cite this version:
Bruno Rossion, Talia Retter, Joan Liu-shuang. Understanding human individuation of unfamiliar
faces with oddball fast periodic visual stimulation and electroencephalography. European Journal of
Neuroscience, Wiley, 2020, 52 (10), pp.4283-4344. �10.1111/ejn.14865�. �hal-02931197�

Rossion, B., Retter, T.L., Liu-Shuang, J. (2020). European Journal of Neuroscience, in press.
1
Understanding human individuation of unfamiliar faces with
oddball fast periodic visual stimulation and electroencephalography
Running Title: Understanding face individuation with oddball FPVS
Bruno Rossion
1,2
, Talia L. Retter
3
, Joan Liu-Shuang
4
1. Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France
2. CHRU-Nancy, Service de Neurologie, F-54000, France
3. Department of Behavioural and Cognitive Sciences, Faculty of Language and
Literature, Humanities, Arts and Education, University of Luxembourg
4. Institute of Research in Psychological Science, Institute of Neuroscience, Université de
Louvain, Belgium
Corresponding author:
Bruno Rossion
CRAN UMR 7039, CNRS - Universite de Lorraine,
2 Avenue de la forêt de Haye
54516 Vandoeuvre-lès-Nancy, France.
Tel: +33 (0)3 83 85 80 53
E-mail: bruno.rossion@univ-lorraine.fr
131 pages, 21 Figures, 1 Table.
Keywords: face individuation, visual face categorization, EEG, unfamiliar faces,
adaptation, frequency-tagging, SSVEP, FPVS

Rossion, B., Retter, T.L., Liu-Shuang, J. (2020). European Journal of Neuroscience, in press.
2
Abstract
To investigate face individuation (FI), a critical brain function in the human species, an oddball
fast periodic visual stimulation (FPVS) approach was recently introduced (Liu-Shuang et al.,
2014). In this paradigm, an image of an unfamiliar “base” facial identity is repeated at a rapid rate
F (e.g., 6 Hz) and different unfamiliar “oddball” facial identities are inserted every n
th
item, at a
F/n rate (e.g., every 5
th
item, 1.2 Hz). This stimulation elicits FI responses at F/n and its harmonics
(2F/n, 3F/n, etc.), reflecting neural discrimination between oddball vs. base facial identities, which
is quantified in the frequency-domain of the electroencephalogram (EEG). This paradigm, used in
20 published studies, demonstrates substantial advantages for measuring FI in terms of validity,
objectivity, reliability, and sensitivity. Human intracerebral recordings suggest that this FI response
originates from neural populations in the lateral inferior occipital and fusiform gyri, with a right
hemispheric dominance consistent with the localization of brain lesions specifically affecting facial
identity recognition (prosopagnosia). Here we summarize the contributions of the oddball FPVS
framework towards understanding FI, including its (a)typical development, with early studies
supporting the application of this technique to clinical testing (e.g., Autism Spectrum Disorder).
This review also includes an in-depth analysis of the paradigm’s methodology, with guidelines for
designing future studies. A large-scale group analysis compiling data across 130 observers provides
insights into the oddball FPVS FI response properties. Overall, we recommend the oddball FPVS
paradigm as an alternative approach to behavioral or traditional event-related-potential EEG
measures of face individuation.

Rossion, B., Retter, T.L., Liu-Shuang, J. (2020). European Journal of Neuroscience, in press.
3
Index
ABSTRACT..................................................................................................................................................................... 2
INDEX.............................................................................................................................................................................. 3
1. INTRODUCTION ................................................................................................................................................ 6
2. THE IMPORTANCE OF HUMAN UNFAMILIAR FACE INDIVIDUATION ............................................ 8
3. MEASURING FACE INDIVIDUATION ........................................................................................................ 10
3.1. THE DESIRED VIRTUES OF A FACE INDIVIDUATION MEASURE ....................................................................................... 10
3.2. THE DIFFICULTIES OF EXPLICIT BEHAVIORAL MEASURES ............................................................................................. 11
3.3. THE DIFFICULTIES OF EVENT-RELATED-POTENTIAL MEASURES OF FACE INDIVIDUATION ..................................... 16
3.4. THE VALUE OF FAST PERIODIC VISUAL STIMULATION .................................................................................................... 18
4. FACE INDIVIDUATION WITH AN ODDBALL FPVS PARADIGM AND EEG .................................... 19
4.1. THE ODDBALL FPVS PARADIGM ....................................................................................................................................... 19
4.2. LARGE-SCALE GROUP ANALYSIS ........................................................................................................................................ 25
5. ADVANTAGES OF THE APPROACH ................................................................................................................ 32
5.1. IS IT A VALID MEASURE OF FACE INDIVIDUATION? ......................................................................................................... 33
5.1.1. Discrimination and generalization, automaticity, and time-constraints ........................................ 33
5.1.2. A high-level FI response despite identical base face images? ............................................................... 34
5.1.3. A high-level FI response: empirical evidence .............................................................................................. 35
5.1.4. Neural specificity and sources .......................................................................................................................... 44
5.1.5. Summary ................................................................................................................................................................... 51
5.2. OBJECTIVITY, SENSITIVITY, RELIABILITY ......................................................................................................................... 51
5.2.1 Objectivity in FI response identification and quantification ................................................................. 51
5.2.2. Sensitivity .................................................................................................................................................................. 54
5.2.3. Reliability .................................................................................................................................................................. 57
5.2.4. Summary ................................................................................................................................................................... 60
6. MECHANISMS ........................................................................................................................................................ 60
6.1. GENERAL NEURAL MECHANISMS OF FREQUENCY-TAGGED EEG RESPONSES ............................................................. 60

Rossion, B., Retter, T.L., Liu-Shuang, J. (2020). European Journal of Neuroscience, in press.
4
6.2. FACE INDIVIDUATION MECHANISMS ................................................................................................................................ 64
7. INSIGHTS INTO FACE INDIVIDUATION ........................................................................................................ 68
7.1 INTERINDIVIDUAL VARIABILITY AND RELATIONSHIP WITH BEHAVIOR ........................................................................ 68
7.2 THE NATURE OF THE FACE INDIVIDUATION PROCESS ..................................................................................................... 72
7.2.1. Shape and surface information ........................................................................................................................ 72
7.2.2. Holistic individuation of faces through the composite face effect ...................................................... 72
7.3. AUTOMATICITY AND TASK-MODULATION ........................................................................................................................ 74
7.4 APPLICATION TO TYPICAL & ATYPICAL DEVELOPMENT ................................................................................................. 76
7.4.1. Typical development............................................................................................................................................. 76
7.4.2. Atypical development ........................................................................................................................................... 82
7.5. TIME-DOMAIN INFORMATION ............................................................................................................................................ 84
7.5.1. How to capture temporal dynamics with this paradigm ....................................................................... 86
7.5.2. Insights into the temporal dynamics of face individuation ................................................................... 86
8. METHODOLOGICAL GUIDELINES ................................................................................................................... 92
8.1. STIMULATION PARAMETERS .............................................................................................................................................. 93
8.1.1. Selecting and controlling the stimulus set ................................................................................................... 93
8.1.2. Frequency selection............................................................................................................................................... 94
8.1.3. Stimulation sequence and overall recording duration ........................................................................... 97
8.1.4. Stimulation presentation mode ....................................................................................................................... 97
8.2. DATA ANALYSIS AND INTERPRETATION ........................................................................................................................... 98
8.2.1. (Pre)processing and robustness of the data ................................................................................................ 98
8.2.2. Multi-harmonic response quantification ...................................................................................................... 99
8.2.3 Interpreting FI response amplitude differences ....................................................................................... 100
9. CONCLUSIONS AND FUTURE APPLICATIONS .......................................................................................... 101
9.1. THE IMPORTANCE OF UNFAMILIAR FACES .................................................................................................................... 102
9.2. FUNCTIONAL PROCESSES ................................................................................................................................................. 104
9.3. IMPLICIT PROCESSING, AWARENESS, AND PERIODICITY ............................................................................................. 105
9.5. TYPICAL AND ATYPICAL DEVELOPMENT ....................................................................................................................... 108
SUPPLEMENTARY MATERIAL: METHODS OF THE LARGE-SCALE GROUP ANALYSIS ................... 110

Frequently asked questions

Q1. What are the contributions mentioned in the paper "Understanding human individuation of unfamiliar faces with oddball fast periodic visual stimulation and electroencephalography" ?

In this paper, the authors used a fast periodic visual stimulation ( FPVS ) mode to measure the frequency of identity change in the human brain. 

Q2. What are the future works mentioned in the paper "Understanding human individuation of unfamiliar faces with oddball fast periodic visual stimulation and electroencephalography" ?

Again, the validity of this measure will have to be continuously evaluated with further studies using the oddball FPVS framework. Future studies could address this issue more systematically, also testing for wider variations of views, expressions, etc. in more natural images, when repeating the same facial identity in the FI oddball paradigm. 9. 2. Functional processes Providing that the methodological factors listed above ( Section 8 ) are taken into consideration, the oddball FPVS-EEG paradigm could prove to be extremely useful in future studies for further clarifying the nature of FI. For example, the authors have mostly used color stimuli in their studies, but how this factor contributes to the FI response is unclear and should be evaluated in future studies. 

Q3. What is the likely explanation of RS effects in the present paradigm?

the present paradigm utilizes stimulus repetitions and changes at short time scales(hundreds of ms), which are appropriate to produce RS effects (which may be produced across many timescales, potentially relating to different neural mechanisms; Grill-Spector et al., 2006; Henson, 2016). 

Q4. What is the drawback of measuring FI with unfamiliar faces?

An obvious drawback of measuring FI with unfamiliar faces is that FI could be based to acertain extent on unnatural, analytical processes and low-level visual cues, which, as noted above, may account for above chance performance at explicit behavioral tests in patients with prosopagnosia, infants, or other animal species (including macaque monkeys), especially in thecontext of long or unlimited presentation durations. 

Q5. What is the likely explanation of FI responses in the present paradigm?

Even though, as mentioned above, an oddball face could generate a decrease, increase or phase shift of neural activity, the authors look to repetition suppression (RS) as the most likely explanation of FI responses in this paradigm for two reasons. 

Q6. How many base face repetitions are advantageous for measuring FI responses?

at present, the authors suspect that having at least three base face repetitions is advantageous for measuring FI responses, based on the decreased F/n response at 1 

Q7. How can changes of input be detected?

Changes of input can be detected at many levels of the neural circuitry, depending not only on the changes in physical input properties but also on the long-term and recent experience of these neural circuits. 

Q8. What is the sensitivity of the FPVS-oddball approach?

the sensitivity of the FPVS-oddball approach allows for a meaningful FI responseto be obtained: 1) in a short amount of testing time; 2) with minimal and straightforward data processing (sometimes without artifact rejection/correction, e.g., Xu et al., 2017); and 3) with significance often attained at the individual subject level, such that results are highly reproducible across studies (see the next section). 

Q9. What is the parsimonious explanation of the FI?

A more parsimonious explanation is that the same population of neurons carries out FI across various change in head views, but is less successful at associating identities when fewer cues match between the unfamiliar face exemplars. 

Q10. How can the authors determine the response significance at the frequencies of interest?

Evaluating response significance at the frequencies-of-interest can be done in astraightforward manner by means of a quantitative/statistical computation relative to the neighboring frequency bins (see Box 1; see also Section 8 for methodological guidelines). 

Q11. Why is it unclear how to evaluate the response in standard ERP paradigms?

Note that in contrast, in standard ERP paradigms, it is unclear how to objectively evaluate responses (e.g., in terms of SNR), due to a lack of objective signal vs. noise definition. 

Q12. How can the authors control for inter-individual variability of other factors?

Although the authors may seldom be conscious of individuating unfamiliar faces in natural settings, the authors nevertheless recognize unfamiliar faces as unfamiliar, and the authors readily notice when this process is challenged,2 Attempts to control for inter-individual variability of other factors can be made by asking participants to run the same behavioral task with another material, e.g., pictures of cars (Dennett et al., 2012) and normalizing the data obtained on faces by the data obtained with pictures of the other material.