Data preparation is a fundamental stage of data analysis. While a lot of low-quality information is available in various data sources and on the Web, many organizations or companies are interested in how to transform the data into cleaned forms which can be used for high-profit purposes. This goal generates an urgent need for data analysis aimed at cleaning the raw data. In this paper, we first show the importance of data preparation in data analysis, then introduce some research achievements in the area of data preparation. Finally, we suggest some future directions of research and development.

/pdf/data-preparation-for-data-mining-x6m99qeujb.pdf

Data preparation for data mining

The letter and/or sound combinations that make up a human language are limited by the human's ability to pronounce tnese sounds° Therefore, the standard library search, which as a rule looks for all possible combinations of letters to find a word, is wasteful. Certain letters simply cannot be followed by certain other letters and a search for them is senseless. Following this same line of reasoning, letters very frequently occur in the combinations that are germane to the particular language. The growing amount of alphanumeric information presently being stored on magnetic tape presents increasingly difficult problems in both the number of tape reels used and the time necessary to search this mass of information in order to extract pertinent literature. At the present time most of this literature on tape utilizes the standard IBM 6-bit code to express alphanumeric symbols. ~t is entirely feasible to record standard English literature on tape -be it professional abstracts or novels -using only approximately two-thirds of the binary bits utilized to represent the same piece of written material in the conventional code. This can be accomplished by setting up, in a 9-bit code, the 400-odd letter combinations occurring most frequently. A 9-bit representation allows the programmer to set up as many as 512 symbols, thus leaving sufficient leeway to assign symbols to the most frequentlyused words, mathematical symbols, professional expressions, that are expected to be encountered in the literature to be recorded. In addition, these relatively short 9-bit symbols can be assigned to all key words that it may be necessary to look for later, thereby accelerating any future library search.

Information storage and retrieval

Data preparation is an important and critical step in neural network modeling for complex data analysis and it has a huge impact on the success of a wide variety of complex data analysis tasks, such as data mining and knowledge discovery. Although data preparation in neural network data analysis is important, some existing literature about the neural network data preparation are scattered, and there is no systematic study about data preparation for neural network data analysis. In this study, we first propose an integrated data preparation scheme as a systematic study for neural network data analysis. In the integrated scheme, a survey of data preparation, focusing on problems with the data and corresponding processing techniques, is then provided. Meantime, some intelligent data preparation solution to some important issues and dilemmas with the integrated scheme are discussed in detail. Subsequently, a cost-benefit analysis framework for this integrated scheme is presented to analyze the effect of data preparation on complex data analysis. Finally, a typical example of complex data analysis from the financial domain is provided in order to show the application of data preparation techniques and to demonstrate the impact of data preparation on complex data analysis.

An integrated data preparation scheme for neural network data analysis

Sentiment analysis of malayalam tweets using machine learning techniques

In today’s world diabetes has become one of the most life threatening and at the same time most common diseases not only in India but around the world. Diabetes is seen in all age groups these days and they are attributed to lifestyle, genetic, stress and age factor. Whatever be the reasons for diabetics, the outcome could be severe if left unnoticed. Currently various methods are being used to predict diabetes and diabetic inflicted diseases. In the proposed work, we have used the Machine Learning algorithms Support Vector Machine (SVM) & Random Forest (RF) that would help to identify the potential chances of getting affected by Diabetes Related Diseases. After pre-processing the data, features which influences the prediction are selected by implementing step forward and backward feature selection. The Principle Component Analysis (PCA) dimensionality reduction method is analyzed after the selection of specific features and the accuracy of the prediction is 83% implementing Random Forest (RF) which is significant in comparison with Support Vector Machine (SVM) with accuracy of 81.4%.

Diabetes Prediction using Machine Learning Algorithms with Feature Selection and Dimensionality Reduction

In view of the exponential growth of online document corpora, even perfect retrieval will fetch too much material for a user to cope with. One way to reduce this problem is automatic domain-specific summarization tailored to user's needs, which is a kind of high-level data cleaning. This requires some method of discovering classes of similar items that may be grouped into predetermined domains. We explore whether there exists a synergic relation between systems for classification and those for summarization by way of composing those subsystems. In other words, we examine whether prior summarization will increase the performance of the classifier system and vice versa. In both cases, the answer is affirmative, as we show in this paper. We propose a text-mining framework in which these subsystems are treated as constituents of a knowledge discovery process for text corpora.

Summarization and categorization of text data in high-level data cleaning for information retrieval

In this paper, Sentence level Sentiment Analysis of Malayalam movie reviews is done by classifying the polarity of opinions obtained from the user as positive, negative and neutral. Sentiment analysis is an application of Natural Language Processing and text analysis which helps to identify the emotions in a given context. In this work a hybrid approach for Sentiment Analysis is used in which Machine Learning method is used for tagging and Fuzzy Logic is used to find the membership of the review in each sentiment class. Fuzzy logic can be used to handle the vagueness in natural language. Manually tagged data are trained using TnT and Fuzzy rules are incorporated for identifying and classifying the emotions. Certain other rules are also incorporated to handle certain special cases. The fuzzy rules yield output that varies in degree between 0 and 1.

Fuzzy logic based hybrid approach for sentiment analysisl of Malayalam movie reviews

Language Identification (LI) is the process of determining the natural language in which the given content is written. It is an important preprocessing step in many tasks of Natural Language Processing (NLP). In a multilingual society like India, automatic language identification has a wider scope, since it would be a vital step in bridging the digital divide between the Indian masses and others. In this paper, we present an N-gram based method of language identification for documents written in Hindi and Sanskrit, which have the same script and the results are shown. The technique can also be applied to other pairs of Indian languages sharing common scripts.

N-gram based algorithm for distinguishing between Hindi and Sanskrit texts

https://cyberleninka.org/article/n/655723.pdf

Unsupervised Approach to Word Sense Disambiguation in Malayalam

Speech/non-speech detection (SND) distinguishes between speech and non-speech segments in recorded audio and video documents. SND systems can help reduce the storage space required when only speech segments from the audio documents are required, for example content analysis, spoken language identification, etc. In this work, we experimented with the use of time domain, frequency domain and cepstral domain features for short time frames of 20 ms. size along with their mean and standard deviation for segments of size 200 ms. We then analysed if selecting a subset of the features can help improve the performance of the SND system. Towards this, we experimented with different feature selection algorithms, and observed that correlation based feature selection gave the best results. Further, we experimented with different decision tree classification algorithms, and note that random forest algorithm outperformed other decision tree algorithms. We further improved the SND system performance by smoothing the decisions over 5 segments of 200 ms. each. Our baseline system has 272 features, a classification accuracy of 94.45 % and the final system with 8 features has a classification accuracy of 97.80 %.

P. C. Reghu Raj

Papers

Summarization and categorization of text data in high-level data cleaning for information retrieval

Fuzzy logic based hybrid approach for sentiment analysisl of Malayalam movie reviews

N-gram based algorithm for distinguishing between Hindi and Sanskrit texts

Unsupervised Approach to Word Sense Disambiguation in Malayalam

Random forest algorithm for improving the performance of speech/non-speech detection