Abstract This paper presents an analysis of face emoji (disc-shaped pictograms with stylized facial expressions) that accompany written text. We propose that there is a use of face emoji in which they comment on a target proposition expressed by the accompanying text, as opposed to making an independent contribution to discourse. Focusing on positively valenced and negatively valenced emoji (which we gloss as happy and unhappy , respectively), we argue that the emoji comment on how the target proposition bears on a contextually provided discourse value endorsed by the author. Discourse values embody what an author desires, aspires to, wishes for, or hopes for. Our analysis derives a range of non-trivial generalizations, including (i) ordering restrictions with regards to the placement of emoji and text, (ii) cases of apparent mixed emotions, and (iii) cases where the lexical content of the accompanying text influences the acceptability of a face emoji. 

/pdf/a-semantics-of-face-emoji-in-discourse-3issi0e3.pdf

A semantics of face emoji in discourse

Language is typically embedded in multimodal communication, yet models of linguistic competence do not often incorporate this complexity. Meanwhile, speech, gesture, and/or pictures are each considered as indivisible components of multimodal messages. Here, we argue that multimodality should not be characterized by whole interacting behaviors, but by interactions of similar substructures which permeate across expressive behaviors. These structures comprise a unified architecture and align within Jackendoff's Parallel Architecture: a modality, meaning, and grammar. Because this tripartite architecture persists across modalities, interactions can manifest within each of these substructures. Interactions between modalities alone create correspondences in time (ex. speech with gesture) or space (ex. writing with pictures) of the sensory signals, while multimodal meaning-making balances how modalities carry “semantic weight” for the gist of the whole expression. Here we focus primarily on interactions between grammars, which contrast across two variables: symmetry, related to the complexity of the grammars, and allocation, related to the relative independence of interacting grammars. While independent allocations keep grammars separate, substitutive allocation inserts expressions from one grammar into those of another. We show that substitution operates in interactions between all three natural modalities (vocal, bodily, graphic), and also in unimodal contexts within and between languages, as in codeswitching. Altogether, we argue that unimodal and multimodal expressions arise as emergent interactive states from a unified cognitive architecture, heralding a reconsideration of the “language faculty” itself.

/pdf/remarks-on-multimodality-grammatical-interactions-in-the-1kjdo5px.pdf

Remarks on Multimodality: Grammatical Interactions in the Parallel Architecture

The interaction effect of emoji and social media content on consumer engagement: A mixed approach on peer-to-peer accommodation brands

ABSTRACT While some argue that emojis are processed like words, opponents note dissimilarities. We used a divided attention (DA) technique to examine whether memory for emojis, relative to words, engages primarily verbal or visuo-spatial cognitive representations. We compared the decline in memory output experienced when participants freely recalled a list of studied words or emojis under dual-task conditions with a concurrently performed distracting task. Participants encoded either words or emojis (between-subjects) under full attention (FA), and later recalled them under FA or while concurrently performing a 1-back task to words (DA Words), emojis (DA Emojis), or stars (DA Stars), manipulated within-subject. Emoji memory was higher overall compared to words. Word recall was unaffected by the DA Stars condition, but significantly worse in the DA Words-and to a lesser degree DA Emojis-condition, relative to FA. Results suggest that memory for words relies primarily on reactivation of verbal representations, which is hampered when the distracting task also requires verbal, but not visuo-spatial, processing. In contrast, small but significant declines in recall were observed for emojis across all DA conditions relative to FA. Results suggest that unlike words, representing and retrieving emojis engages both verbal and visuo-spatial processing.

Are emojis processed visuo-spatially or verbally? Evidence for dual codes

https://doi.org/10.1016/j.dib.2022.108399

Marie de la Fuente

Papers

The processing of emoji-word substitutions: A self-paced-reading study

Stimulus data and experimental design for a self-paced reading study on emoji-word substitutions