Social media like Twitter have become very popular in recent decades Hashtags are new kind of metadata which make non-structured tweets into searchable semistructured content There are varied previous methods which recommend hashtags for new tweets However, to the best of our knowledge, there is no previous word that uses BERT embedding for this purpose In this paper, we propose a new method called EmHash that uses neural network based on BERT embedding to recommend new hashtags for each tweet Unlike other word embeddings, BERT embedding constructs different vectors for the same word in different contexts Emhash succeeded in outperforming three methods LDA, SVM, and TTM with respect to recall measure

EmHash: Hashtag Recommendation using Neural Network based on BERT Embedding

Corpus based Machine Translation System with Deep Neural Network for Sanskrit to Hindi Translation

In modern age of information explosion, every day millions of gigabytes of data are generated in the form of documents, web pages, e-mail, social media text, blogs etc., so importance of effective and efficient Natural Language Processing techniques become crucial for an information retrieval system, text summarization, sentiment analysis, information extraction, named entity recognition, relationship extraction, social media monitoring, text mining, language translation program, and question answering system. Natural Language Processing is a computational technique applies different levels of linguistic analysis for representing natural language into a useful representation for further processing. NLP is recognized as a challenging task in computer science and artificial intelligence because understanding human natural language is not only depends on the words but how those words are linked together to form precise meaning is also considered. Regardless of language being one of the easiest concepts for human to learn, but for training computers to understand natural language is a difficult task due to the ambiguity of language syntax and semantics. Natural Language processing techniques involves processing documents or text which reduces storage space and also reduces the size of index and understanding the given information which satisfies user’s need. NLP techniques improve the performance of the information retrieval efficiency and effective documentation processes. Common dialect handling procedures incorporates tokenization, stop word expulsion, stemming, lemmatization, parts of discourse labeling, lumping and named substance recognizer which enhances execution of NLP applications. The Natural Language Toolkit is the best possible solution for learning the ropes of NLP domain. NLTK, a collection of application packages which encourage researchers and learners in natural language processing, computational linguistics and artificial intelligence.

Review on Natural Language Processing Trends and Techniques Using NLTK.

In the era of information burst, optimization of processes for Information Retrieval, Text Summarization, Text and Data Analytic systems becomes utmost important. Therefore in order to achieve accuracy, redundant words with low or no semantic meaning must be filtered out. Such words are known as Stopwords. Stopwords list has been developed for languages like English, Chinese, Arabic, Hindi, etc but standard stopword list is still missing for Sanskrit language. Identifying stop words manually from Sanskrit text is a herculean task hence this paper reflects an automated stop word generator algorithm based on frequency of word and its implementation to ease the task. To fine-tune the generated list still manual intervention by language expert is required thus following a hybrid approach. The paper presents the first of its kind, a list of seventy-five generic stopwords of Sanskrit language extracted from a data amounting to nearly seventy-six thousand words.

Generating Stopword List for Sanskrit Language

The work in this paper presents a MTS from Sanskrit to English language using a hybridized form of direct and rule-based machine translation technique. This paper also discusses the language divergence among Sanskrit and English languages with a recommended solution to handle the divergence. The proposed system has used two bilingual dictionaries (Sanskrit–English, Sanskrit–UNL), a tagged Sanskrit corpus, a Sanskrit analysis rule base and an ELGR base. Elasticsearch technique has enhanced the translation speed of the proposed system for accessing the data from different data dictionaries and rule bases used for the system development. The system uses CFG in CNF for Sanskrit language processing and CYK parsing technique for processing the input Sanskrit sentence. This work also presents a novel algorithm which creates a parse tree from the parsing table. ELGR base and bilingual dictionaries generate the target language sentence. The proposed system is evaluated using natural language toolkit API in python and achieved a BLEU score of 0.7606, fluency score of 3.63 and adequacy score of 3.72. A comparison of the proposed system with state-of-the-art systems shows that the proposed system outperforms existing systems.

https://link.springer.com/content/pdf/10.1007/s00521-020-05156-3.pdf

A Sanskrit-to-English machine translation using hybridization of direct and rule-based approach

In the Information era, optimization of processes for Information Retrieval, Text Summarization, Text and Data Analytic systems becomes utmost important. Therefore in order to achieve accuracy, extraction of redundant words with low or no semantic meaning must be filtered out. Such words are known as stopwords. Stopwords list has been developed for languages like English, Chinese, Arabic, Hindi, etc. Stopword list is also available for Sanskrit language. Stop-word removal is an important preprocessing techniques used in Natural Language processing applications so as to improve the performance of the Information Retrieval System, Text Analytics & Processing System, Text Summarization, Question-Answering system, stemming etc. In this paper, a simple approach is used to design stop-word removal algorithm and its implementation for Sanskrit language. The algorithm and its implementation uses dictionary based approach. In dictionary based approach predefined list of stopwords is compared to the target text on which removal is required.

/pdf/stop-word-removal-algorithm-and-its-implementation-for-229kb3nxcm.pdf

Stop-Word Removal Algorithm and its Implementation for Sanskrit Language

Machine Translation is area of research since six decades. It is gaining popularity since last decade due to better computational facilities available at personal computer systems. This paper presents different Machine Translation system where Sanskrit is involved as source, target or key support language. Researchers employ various techniques like Rule based, Corpus based, Direct for machine translation. The main aim to focus on Sanskrit in Machine Translation in this paper is to uncover the language suitability, its morphology and employ appropriate MT techniques.

/pdf/sanskrit-machine-translation-systems-a-comparative-analysis-1gb2tgvten.pdf

Sanskrit Machine Translation Systems: A Comparative Analysis

The process of stripping off affixes from a word to arrive at root word or lemma is known as Lemmatization. The usefulness of lemmatizer in natural language operations cannot be overlooked especial...

Sanskrit lemmatizer for improvisation of morphological analyzer

In linguistics, morphology is a study regarding word, word formation, its analysis, and generation. A morphological analyzer is a tool to understand grammatical characteristics and constituent’s part-of-speech information. A morphological analyzer is a useful tool in many NLP implementations such as syntactic parser, spell checker, information retrieval, and machine translation. Here, 328 Sanskrit words are tested through four morphological analyzers namely—Samsaadhanii, morphological analyzers by JNU and TDIL, both of which are available online and locally developed and installed Sanguj morphological analyzer. There is a negligible divergence in the reflected results.

Jaideepsinh K. Raulji

Papers

Stop-Word Removal Algorithm and its Implementation for Sanskrit Language

Generating Stopword List for Sanskrit Language

Sanskrit Machine Translation Systems: A Comparative Analysis

Sanskrit lemmatizer for improvisation of morphological analyzer

Peer Analysis of “Sanguj” with Other Sanskrit Morphological Analyzers