Language and the Internet

Code-mixing is frequently observed in user generated content on social media, especially from multilingual users. The linguistic complexity of such content is compounded by presence of spelling variations, transliteration and non-adherance to formal grammar. We describe our initial efforts to create a multi-level annotated corpus of Hindi-English codemixed text collated from Facebook forums, and explore language identification, back-transliteration, normalization and POS tagging of this data. Our results show that language identification and transliteration for Hindi are two major challenges that impact POS tagging accuracy.

/pdf/pos-tagging-of-english-hindi-code-mixed-social-media-content-wzbx0d4taj.pdf

POS Tagging of English-Hindi Code-Mixed Social Media Content

Hate speech detection in social media texts is an important Natural language Processing task, which has several crucial applications like sentiment analysis, investigating cyberbullying and examining socio-political controversies. While relevant research has been done independently on code-mixed social media texts and hate speech detection, our work is the first attempt in detecting hate speech in Hindi-English code-mixed social media text. In this paper, we analyze the problem of hate speech detection in code-mixed texts and present a Hindi-English code-mixed dataset consisting of tweets posted online on Twitter. The tweets are annotated with the language at word level and the class they belong to (Hate Speech or Normal Speech). We also propose a supervised classification system for detecting hate speech in the text using various character level, word level, and lexicon based features.

/pdf/a-dataset-of-hindi-english-code-mixed-social-media-text-for-3nemp5d54y.pdf

A Dataset of Hindi-English Code-Mixed Social Media Text for Hate Speech Detection.

Language is a social phenomenon and variation is inherent to its social nature. Recently, there has been a surge of interest within the computational linguistics CL community in the social dimension of language. In this article we present a survey of the emerging field of "computational sociolinguistics" that reflects this increased interest. We aim to provide a comprehensive overview of CL research on sociolinguistic themes, featuring topics such as the relation between language and social identity, language use in social interaction, and multilingual communication. Moreover, we demonstrate the potential for synergy between the research communities involved, by showing how the large-scale data-driven methods that are widely used in CL can complement existing sociolinguistic studies, and how sociolinguistics can inform and challenge the methods and assumptions used in CL studies. We hope to convey the possible benefits of a closer collaboration between the two communities and conclude with a discussion of open challenges.

https://biblio.ugent.be/publication/8694686/file/8698904.pdf

Computational sociolinguistics: A survey

Sentiment analysis (SA) using code-mixed data from social media has several applications in opinion mining ranging from customer satisfaction to social campaign analysis in multilingual societies. Advances in this area are impeded by the lack of a suitable annotated dataset. We introduce a Hindi-English (Hi-En) code-mixed dataset for sentiment analysis and perform empirical analysis comparing the suitability and performance of various state-of-the-art SA methods in social media. In this paper, we introduce learning sub-word level representations in our LSTM (Subword-LSTM) architecture instead of character-level or word-level representations. This linguistic prior in our architecture enables us to learn the information about sentiment value of important morphemes. This also seems to work well in highly noisy text containing misspellings as shown in our experiments which is demonstrated in morpheme-level feature maps learned by our model. Also, we hypothesize that encoding this linguistic prior in the Subword-LSTM architecture leads to the superior performance. Our system attains accuracy 4-5% greater than traditional approaches on our dataset, and also outperforms the available system for sentiment analysis in Hi-En code-mixed text by 18%.

Towards Sub-Word Level Compositions for Sentiment Analysis of Hindi-English Code Mixed Text

Code-Mixing is a frequently observed phenomenon in social media content generated by multi-lingual users. The processing of such data for linguistic analysis as well as computational modelling is challenging due to the linguistic complexity resulting from the nature of the mixing as well as the presence of non-standard variations in spellings and grammar, and transliteration. Our analysis shows the extent of Code-Mixing in English-Hindi data. The classification of Code-Mixed words based on frequency and linguistic typology underline the fact that while there are easily identifiable cases of borrowing and mixing at the two ends, a large majority of the words form a continuum in the middle, emphasizing the need to handle these at different levels for automatic processing of the data.

/pdf/a-oei-am-borrowing-ya-mixing-a-an-analysis-of-english-hindi-w0ofjvv4wr.pdf

â€œI am borrowing ya mixing ?â€ An Analysis of English-Hindi Code Mixing in Facebook

In this paper, we target the problem of word level language identication for Indian languages written in roman script and mixed with English language. In addition to the language identication we also handle transliteration of Indian language words into the native Indic scripts. To address these problems we consider a supervised approach of building a classier with monolingual samples together with a context-switching probability from Indian Language (IL) to English (Eng). The proposed system is submitted to the FIRE-2013 shared task on Transliterated Search. Our submitted system showed the best performing results.

https://www.isical.ac.in/~fire/wn/STTS/2013_translit_search-gella-msri.pdf

Query word labeling and Back Transliteration for Indian Languages: Shared task system description

In the recent years, petabytes of data is being generated and uploaded online every second. To successfully detect fake contents, a deepfake detection technique is used to determine whether the uploaded content is real or fake. In this paper, a convolutional neural network-based model is proposed to detect the fake face images. The generative adversarial networks and data augmentation are used to generate the face dataset for real and fake face classification. Transfer learning techniques from pretrained deep models such as VGG16 and ResNet50 are employed in the proposed model. The proposed model is evaluated on three benchmark datasets, namely 140k Real and Fake Faces, Real and Fake Face Detection, and Fake Faces. The proposed model attained accuracies over the three datasets are 95.85%, 53.25%, and 88.63%, respectively. Moreover, to improve the obtained results of the proposed model, we combine it with other pretrained models of VGG16 and ResNet50 to construct deep ensembles. The overall performance is greatly improved with the ensemble model achieving accuracies on the three datasets as 98.79%, 75.79%, and 95.52%, respectively. Furthermore, the obtained results also show that the proposed models have superior performance than existing models.

Deepfakes Classification of Faces Using Convolutional Neural Networks

In this paper, we present a Local Histogram Equalization (LHE) based method to improve the visual quality of foggy images. From the fog-degraded image we separate all three color components and apply the enhancement algorithm to each of these components. To mitigate the issue of blocking artifacts generated from Local Histogram Equalization with non-overlapping blocks, we propose to process blocks by overlapping them. Processing smaller block size yields better visual quality at the expense of relatively higher processing time, therefore, we also propose to use statistically optimal block size to obtain a good visual quality in reasonable time duration.

Jatin Sharma

Papers

POS Tagging of English-Hindi Code-Mixed Social Media Content

â€œI am borrowing ya mixing ?â€ An Analysis of English-Hindi Code Mixing in Facebook

Query word labeling and Back Transliteration for Indian Languages: Shared task system description

Deepfakes Classification of Faces Using Convolutional Neural Networks

Two-Pass Adaptive Histogram based method for Restoration of Foggy Images