Text categorization-assignment of natural language texts to one or more predefined categories based on their content-is an important component in many information organization and management tasks.Different automatic learning algorithms for text categori-zation have different classification accuracy.Very accurate text classifiers can be learned automatically from training examples.

Inductive learning algorithms and representations for text categorization

In this report, the results of my internship about the data analysis of environmental DNA are presented. Environmental DNA is a new technique that gives a better insight which factors determine the water quality of lakes. This technique is being developed by Witteveen+Bos and Datura. Cluster analysis was used to check if the environmental DNA profiles of the lakes could be used to categorize the lakes. There were several methods considered. Namely, the dissimilarity coefficients: Euclidean, Correlation, Bray-Curtis and Jaccard were used in conjunction with the linkage methods: Single, Complete, Average, Weighted and Ward. In order to find the best method for this type of dataset the clustering results of the methods with several filters on the dataset were being compared.
This was done to check for the robustness of the methods. Principal component analysis was used in order to visualize the clustering results and to see if there were there were factors that are characteristic for a cluster.

Statistical learning methods for environmental DNA

The invention relates to a natural language processing technology, in particular to a method for constructing a classification corpus by means of the Internet, comprising the following steps: S1, setting a target class; 2, setting an information source; S3, performing information source rating; S4, performing information collection and analysis; S5, filtering the backup document corpus set; S6, outputting classification corpus. A substantial effect of the present invention is that: on the premise of minimizing the level of manual intervention, based on the Internet pages with clear classification logo and dynamically updated list content as the corpus information source, combined with web page typesetting characteristics and web page DOM node topological structure characteristics, the accuracy of web page topic corpus extraction is improved. Through the matching evaluation system of target categories and information source keywords, the Internet dynamic corpus is screened by quantitative similarity between texts, and a high-quality text classification corpus is constructed.

A method and a system for constructing a classification corpus by means of the Internet

The invention provides a method and a system for data classification. The method comprises: obtaining correlation degree sets of various classification fields, the set being generated in advance, wherein the correlation degree set of each classification field includes a name word of a corresponding field, keywords related to the field represented by the name word of the field, and a first correlation degree of each keyword and the name word of the field; aimed at the correlation degree set of each classification field, determining second correlation degrees of a to-be-classified word and preset related words in the correlation degree set of the classification field, wherein the preset related words include each keyword corresponding to the classification field, or the field name word corresponding to the classification field; inputting the first correlation degrees and the second correlation degrees to a classification field determination model which is established in advance, to obtain the correlation degree of the to-be-classified word and the classification field; according to the correlation degrees of the to-be-classified word and multiple classification fields, determining aclassification result.

Method and system for data classification

The invention provides a dialogue generation method, a dialogue generation device, an electronic device and a computer readable storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining an input statement; determining the relevancy between the input statement and a preset corpus; if the relevancy between the input statement and the preset corpus reaches a preset value, processing the input statement based on a dialogue generation model to generate a reply statement of the input statement; wherein the dialogue generation model isa machine learning model obtained by training sample statements in the preset corpus in advance. According to the method and the device, the reply statement close to the intention of the user can be given in the specific corpus or field, the dialogue quality is improved, the operand of the processing process is reduced, and the efficiency is improved.

Dialogue generation method, dialogue generation device, electronic device and storage medium

The invention provides a classified corpus establishing method and system and a server provided with the system. The establishing method comprises the steps of acquiring target data to be classified and acquiring category description data according to actual needs, selecting a text similarity calculating method corresponding to maximum accuracy, classifying the target data to be classified as a category corresponding to maximum similarity, filling the target data with first classification matching degree within a first similarity range in a preset primary corpus, classifying the rest of the target data to be classified with a selected and well trained classifier, filling the target data with second classification matching degree within a second similarity range in the preset primary corpus, and determining the preset primary corpus as a final corpus when the filled preset primary corpus can not be enlarged any more. In this way, corpus establishment cost is reduced, manual intervene degree is reduced, and corpus establishment time is shortened.

Classified corpus establishing method and system and server provided with system

With the rapid development of online recruitment, a large amount of job postings enables a new paradigm for studying the national economic state. However, there are two problems when using these data for national economic research. First, there is a mismatch between the job descriptions and the job titles. Second, these titles differ from the category names in the national economic research standards. To map job postings with similar job descriptions but different titles to the same category of the national economic research standards, this paper introduces a text classification corpus named HR-CTC. A novel method to reduce the influence of human subjectivity is proposed to construct the corpus. The experimental results show that the proposed method outperforms manual categorization in accuracy evaluation criteria by a 47.08% increase. To verify the value of this corpus for text classification research and to provide a baseline for further research, we implement five methods of deep learning for text classification and achieve promising results.

HR-CTC: A Large Human Resource Corpus for Text Classification.

The absence of an appropriate text classification corpus makes the massive amount of online job information unusable for labor market analysis This paper presents JCTC, a large job posting corpus for text classification In JCTC construction framework, a formal specification issued by the Chinese central government is chosen as the classification standard The unsupervised learning (WE-cos), supervised learning algorithm (SVM) and human judgements are all used in the construction process JCTC has 102581 online job postings distributed in 465 categories The method proposed here can not only ameliorate the high demands on people's skill and knowledge, but reduce the subjective influences as well Besides, the method is not limited in Chinese We benchmark five state-of-the-art deep learning approaches on JCTC providing baseline results for future studies JCTC might be the first job posting corpus for text classification and the largest one in Chinese With the help of JCTC, related organizations are able to monitor, analyze and predict the labor market in a comprehensive, accurate and timely manner

Chongyang Gu

Papers

Classified corpus establishing method and system and server provided with system

HR-CTC: A Large Human Resource Corpus for Text Classification.

JCTC: A Large Job posting Corpus for Text Classification