Fault tolerance in big data storage and processing systems: A review on challenges and solutions

Algorithm stemming of Nazief and Andriani has been developed in terms of speed and accuracy. One of the developments is Non-Formal Affix Algorithm. This algorithm increases the accuracy of stemming in non-formal affixed words. In its development, Indonesian language is used in two forms: standard and non-standard. Non-formal language is commonly used in non-formal situation such as in daily conversation, post, or comment on social media (Facebook, Twitter, Instagram, etc.). Non-formal Affix Stemming Algorithm is used to get the basic word of a non-formal word. However, this algorithm has limitation in non-formal affixed word stemming. Thus, this research aims to focus on the modification of non-formal affix algorithm to increase accuracy in non-formal affixed words stemming. The result of this research shows that modification of the combination of Levenshtein Distance and frequently used words dictionary has accuracy level of 88.33% with precision of 1 and error rate of 0.2. Meanwhile, Non-formal Affix algorithm has accuracy level of 73.3% at stemming 60 non-formal affixed words with precision of 1 and error rate of 0.26. Therefore, it can be concluded that Lenshtein Distance approach can increase the accuracy of Non-formal Affix algorithm in non-formal affixed words stemming.

Accuracy Measurement on Indonesian Non-formal Affixed Word Stemming With Levenhstein

Cost-aware load balancing for multilingual record linkage using MapReduce

The data linkage techniques “entity linking” and “record linkage” get rising attention as they enable the integration of multiple data sources for data, web, and text mining approaches. This has resulted in the development of numerous algorithms and systems for these techniques in recent years. The goal of this publication is to provide an overview of these numerous data linkage techniques. Most papers deal with record linkage and structured data. Processing unstructured data through entity linking is rising attention with the trend Big Data. Currently, deep learning algorithms are being explored for both linkage techniques. Most publications focus their research on a single process step or on the entire process of “entity linking” or “record linkage”. However, the papers have the limitation that the used approaches and techniques have always been optimized for only a few data sources.

/pdf/a-qualitative-literature-review-on-linkage-techniques-for-38bmycq3tj.pdf

A Qualitative Literature Review on Linkage Techniques for Data Integration

Stemming algorithm Nazief & Andriani has been development in terms of the speed and the accuracy. One of its development is Non-formal Affix Algorithm. Non-formal Affix Algorithm improves the accuracy for non-formal affixed word. In its growth, Indonesian language is used in two ways: formal and non-formal. Non-formal language is commonly used in casual situations such as conversations and social media post (Facebook, Twitter, Instagram, etc.). To get the root of the word of a casual conversation or a social media post, stemming algorithm which can process the non-formal words with affixes already proposed. But, the previous algorithm unable to stem a non-formal word that slightly change the root word. Therefore, this study modifies Non-formal Affix Algorithm to increase stemming accuracy on non-formal word. Modifications are made by adding Levenshtein Distance. The result of the research shows that the algorithm made in this research has 96.6% accuracy while the Non-formal Affix algorithm has 73.3% accuracy in processing 60 non-formal affixed words. Based on the result, Levenshtein Distance approach can increase the accuracy on stemming non-formal affixed word.

/pdf/optimalisasi-stemming-kata-berimbuhan-tidak-baku-pada-bahasa-58blud8yxp.pdf

Optimalisasi Stemming Kata Berimbuhan Tidak Baku Pada Bahasa Indonesia Dengan Levenshtein Distance

Name matching is a key component in various applications in our life like record linkage and data mining applications. This process suffers from multiple complexities such as matching data from different languages or data written by people from different cultures. In this paper, we present a new modified Cross-Language Levenshtein Distance (CLLD) algorithm that supports matching names across different writing scripts and with many-to-many characters mapping. In addition, we present a hybrid cross-language name matching technique that uses phonetic matching technique mixed with our proposed CLLD algorithm to improve the overall f-measure and speed up the matching process. Our experiments demonstrate that this method substantially outperforms a number of well-known standard phonetic and approximate string similarity methods in terms of precision, recall, and f-measure.

Doaa Medhat

Papers

A hybrid cross-language name matching technique using novel modified Levenshtein Distance

Cost-aware load balancing for multilingual record linkage using MapReduce