Word similarity is broadly used in many applications, such as information retrieval, information extraction, text classification, word sense disambiguation, example -based machine translation, etc. There are two different methods used to compute similarity: one is based on ontology or a semantic taxonomy; the other is based on collocations of words in a corpus. As a lexical knowledgebase with rich semantic information, How-net has been employed in various researches. Unlike other thesauri, such as WordNet and Tongyici Cilin, in which word similarity is defined based on the distance between words in a semantic taxonomy tree, How-net defines a word in a complicated multi-dimensional knowledge description language. As a result, a series of problems arise in the process of word similarity computation using How-net. The difficulties are outlined below: 1. The description of each word consists of a group of sememes. For example, the Chinese word “暗箱(camera obscura)” is described as: “part|部件, #TakePicture|拍攝, %tool|用具 , body|身”, and the Chinese word “寫信 (write a letter)” is described as: “write|寫, ContentProduct=letter|信件”; 2. The meaning of a word is not a simple combination of these sememes. Sememes are organized using a specific knowledge description language. To meet these challenges, our work includes: 1. A study on the How-net knowledge description language. We rewrite the How-net definition of a word in a more structural format, using the abstract data structure of set and feature structure. 2. A study on the algorithm used to compute word similarity based on How-net. The similarity between sememes, that between sets , and that between feature structures are given. To compute the similarity between two sememes, we

/pdf/ji-yu-zhi-wang-de-ci-hui-yu-yi-xiang-si-du-ji-suan-word-3q2xf1mp2c.pdf

基於《知網》的辭彙語義相似度計算 (Word Similarity Computing Based on How-net).

A robust zero-watermarking algorithm is proposed based on merging features of sentences for Chinese text document authentication. In the scheme, a text is first segmented into sets of sentences, where a semantic code for every word can be obtained. Then the sentence entropy is calculated by the frequency of semantic codes, and the sentence relevance is calculated by the semantic similarity between words through the tree structure of words in Tongyici Cilin. By employing the sentence entropy, the sentence relevance, and the sentence length, a weighting function is used to obtain the final weight of each sentence. The nouns and verbs of the high weight sentences are selected to construct a watermark, which is encrypted and registered with a trusted third party called Certificate Authority (CA). To resolve disputes, the similarity between the watermark generated from the suspicious text and the watermark from CA is calculated. The experimental results show that the proposed algorithm offers better performanc...

A zero-watermarking algorithm based on merging features of sentences for Chinese text

The abundant aspects of big data and it's technology are increasing due to new methods of fetching data and diverse needs. Meteorological data is also the source of big data in terms of volume, variety, veracity and velocity, and it includes structured, unstructured and hybrid forms. This paper aims to apply Hadoop architecture and MapReduce algorithm into meteorological big data. It also describes the difficult situations of dealing complex meteorological data by present computer systems and how we replace it with distributed computing. Automatic text classification is used to divide the text into predefined classes, which help individuals utilize information with an effective method. This paper gives an idea about how to implement meteorological text classification through big data approach. Traditional text classification requires a high cost of domain knowledge, but it has low classification efficiency, which can't satisfy the requirement of large documents processing. Distributed parallel method is well developed to solve the problem of a large number of documents classification. Finally, this paper describes big data applications of meteorological big data combined with agriculture, transportation and power grid industries, and illustrates how we analyze tourism industry by our meteorological tourism big data platform.

Application of meteorological big data

The progressive increase of information content has recently made it necessary to create a system for automatic classification of documents. In this article, a system is presented for the categorization of multiclass Farsi documents that requires fewer training examples and can help to compensate the shortcoming of the standard training dataset. The new idea proposed in the present article is based on extending the feature vector by adding some words extracted from a thesaurus and then filtering the new feature vector by applying secondary feature selection to discard inappropriate features. In fact, a phase of secondary feature selection is applied to choose more appropriate features among the features added from a thesaurus to enhance the effect of using a thesaurus on the efficiency of the classifier. To evaluate the proposed system, a corpus is gathered from the Farsi Wikipedia website and some articles in the Hamshahri newspaper, the Roshd periodical, and the Soroush magazine. In addition to studying the role of a thesaurus and applying secondary feature selection, the effect of a various number of categories, size of the training dataset, and average number of words in the test data also are examined. As the results indicate, classification efficiency improves by applying this approach, especially when available data is not sufficient for some text categories. © 2011 Wiley Periodicals, Inc.

Improving Farsi multiclass text classification using a thesaurus and two-stage feature selection

With the growing amount of textual information available on the Internet, the importance of automatic text classification has been increasing in the last decade. In this paper, a system was presented for the classification of multi-class Farsi documents which uses Support Vector Machine (SVM) classifier. The new idea proposed in the present paper, is based on extending the feature vector by adding some words extracted from a thesaurus. The goal is to assist classifier when training dataset is not comprehensive for some categories. For corpus preparation, Farsi Wikipedia website and articles of some archived newspapers and magazines are used. As the results indicate, classification efficiency improves by applying this approach. 0.89 micro F-measure were achieved for classification of 10 categories of Farsi texts.

https://link.springer.com/content/pdf/10.1007/978-3-642-19437-5_20.pdf

Using thesaurus to improve multiclass text classification

To overcome the limitations of traditional text classification approaches based on bag-of-words representation and to effectively incorporate linguistic knowledge and conceptual index into text vector space model, based on two thesaurus HowNet and Tongyici Cilin(hereinafter referred to Cilin), we use semantic vector to describe a document instead of traditional keywords vector, which is based on merging words with high similarity and using a concept to describe the semantic feature rather than a series of words It not only reduces feature dimension but also adds semantic information to the vector We also use sentence (document) similarity based on simple vector distance to classify the text and three groups of experiments are made respectively The results show that the accuracy rates are generally improved along with semantic treatment, which indicates that semantic mining is very important and necessary to text classification

Research of Chinese Text Classification Methods Based on Semantic Vector and Semantic Similarity

With the Internet developing, there are many documents on the same topic which contains redundance. Multi-Document summarization is a technology of natural languages processing, which extract important information from multiple texts about the same topic according to the compression ratio. Sentence selection is an important part of Multi-document summarization .In this paper, we design a calculate method for Chinese words semantic similarity based on Hownet and Tongyici Cilin and we also design a calculate method for sentences semantic similarity based on N-gram. Using these methods, we evaluate the importance of each candidate sentence based on exploiting both the feature of correlation with the query and the feature of the global association feature. We use an improved MMR method to select the sentence in order to reducing the redundancy. The evaluation and results are presented, which prove that the proposed methods are efficient and the summaries generated are good.

Jing-min Zhou

Papers

Research of Chinese Text Classification Methods Based on Semantic Vector and Semantic Similarity

A sentence selection method of query-based Chinese Multi-document summarization