The article is devoted to the problem of the unity and integrity of the image of a semantic standard, allocated by phrases for a topical text. Herewith, the proximity of the text to the standard is assessed without searching for paraphrases, and the base for assessing the proximity of the text to the standard is the division of words of each of its phrases into classes according to the value of the TF-IDF measure relative to the texts of the corpus, previously formed by the expert. The analyzed texts are abstracts of scientific articles together with their titles. The core of the problem is as follows: for each phrase, the maximum proximity to the standard is achieved with respect to its corpus document and, as a consequence, it is required to assess the mutual relevance of such documents for different phrases of the analyzed text. In this study, this problem is solved by introducing the distances between the vectors of the values of the TF-IDF measure of the words of a separate phrase with respect to different documents in the corpus. In this case, the distance between documents, relative to which the closest proximity to the standard of phrases of the analyzed text was achieved, should be minimal. Using the Euclidean metric and Manhattan distance as an example, this study illustrates the application of the proposed approach to the problem of choosing a higher-level text for the given one in the hierarchy being formed in terms of semantic standard complementarity.

Additive regularizarion of topic models with fast text vectorizartion

Analysis of the Mutual Relevance of Topical Corpus Documents in the Problem of Assessing the Proximity of Text to the Semantic Standard

The central problem solved in the article is the task of automating the process of revealing the information content of the media text. This problem is relevant, since its solution allows you to identify texts that do not carry useful information, but try to influence the reader in order to encourage him to take various actions. First, we present a modification of the CBOW algorithm and describe the process of training a neural network to build word embedding. Next, we describe the Russian word embeddings available on the site https://rusvectores.org/ru/models/ . Here we also show how ready-made word embeddings can be used to solve problems of thematic classification of texts. To do this, we use a set of news data from the resource https://lenta.ru . Finally, to solve the problem of information content of texts, we train the embedding of words based on the given set of texts and conduct testing. The quality check of the model is carried out on the basis of a dataset prepared by the employees of the Volgograd State University. 

Machine Learning Methods and Words Embeddings in the Problem of Identification of Informative Content of a Media Text

/pdf/additive-regularizarion-of-topic-models-with-fast-text-4ppmtu032g.pdf

Additive regularizarion of topic models with fast text vectorizartion

Citations

Analysis of the Mutual Relevance of Topical Corpus Documents in the Problem of Assessing the Proximity of Text to the Semantic Standard

Machine Learning Methods and Words Embeddings in the Problem of Identification of Informative Content of a Media Text

Related Papers (5)

Additive Regularization for Topic Models of Text Collections

Mining Ethnic Content Online with Additively Regularized Topic Models

Topic models can improve domain term extraction

The Development of Topic Models in Natural Language Processing

Short Text Topic Modeling with Flexible Word Patterns