Research aimed at correcting words in text has focused on three progressively more difficult problems:(1) nonword error detection; (2) isolated-word error correction; and (3) context-dependent work correction. In response to the first problem, efficient pattern-matching and n-gram analysis techniques have been developed for detecting strings that do not appear in a given word list. In response to the second problem, a variety of general and application-specific spelling correction techniques have been developed. Some of them were based on detailed studies of spelling error patterns. In response to the third problem, a few experiments using natural-language-processing tools or statistical-language models have been carried out. This article surveys documented findings on spelling error patterns, provides descriptions of various nonword detection and isolated-word error correction techniques, reviews the state of the art of context-dependent word correction techniques, and discusses research issues related to all three areas of automatic error correction in text.

Techniques for automatically correcting words in text

A huge number of informal messages are posted every day in social network sites, blogs, and discussion forums. Emotions seem to be frequently important in these texts for expressing friendship, showing social support or as part of online arguments. Algorithms to identify sentiment and sentiment strength are needed to help understand the role of emotion in this informal communication and also to identify inappropriate or anomalous affective utterances, potentially associated with threatening behavior to the self or others. Nevertheless, existing sentiment detection algorithms tend to be commercially oriented, designed to identify opinions about products rather than user behaviors. This article partly fills this gap with a new algorithm, SentiStrength, to extract sentiment strength from informal English text, using new methods to exploit the de facto grammars and spelling styles of cyberspace. Applied to MySpace comments and with a lookup table of term sentiment strengths optimized by machine learning, SentiStrength is able to predict positive emotion with 60.6p accuracy and negative emotion with 72.8p accuracy, both based upon strength scales of 1–5. The former, but not the latter, is better than baseline and a wide range of general machine learning approaches. © 2010 Wiley Periodicals, Inc.

/pdf/sentiment-in-short-strength-detection-informal-text-4evpqts817.pdf

Sentiment in short strength detection informal text

This paper provides an overview of methods and systems developed for record linkage. Modern record linkage begins with the pioneering work of Newcombe and is especially based on the formal mathematical model of Fellegi and Sunter. In their seminal work, Fellegi and Sunter introduced many powerful ideas for estimating record linkage parameters and other ideas that still influence record linkage today. Record linkage research is characterized by its synergism of statistics, computer science, and operations research. Many difficult algorithms have been developed and put in software systems. Record linkage practice is still very limited. Some limits are due to existing software. Other limits are due to the difficulty in automatically estimating matching parameters and error rates, with current research highlighted by the work of Larsen and Rubin.

/pdf/the-state-of-record-linkage-and-current-research-problems-11n647tbz4.pdf

The State of Record Linkage and Current Research Problems

The problem of merging multiple databases of information about common entities is frequently encountered in KDD and decision support applications in large commercial and government organizations. The problem we study is often called the Merge/Purge problem and is difficult to solve both in scale and accuracy. Large repositories of data typically have numerous duplicate information entries about the same entities that are difficult to cull together without an intelligent ’’equational theory‘‘ that identifies equivalent items by a complex, domain-dependent matching process. We have developed a system for accomplishing this Data Cleansing task and demonstrate its use for cleansing lists of names of potential customers in a direct marketing-type application. Our results for statistically generated data are shown to be accurate and effective when processing the data multiple times using different keys for sorting on each successive pass. Combing results of individual passes using transitive closure over the independent results, produces far more accurate results at lower cost. The system provides a rule programming module that is easy to program and quite good at finding duplicates especially in an environment with massive amounts of data. This paper details improvements in our system, and reports on the successful implementation for a real-world database that conclusively validates our results previously achieved for statistically generated data.

Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem

In this article we propose a strategy for the summarization of scientific articles that concentrates on the rhetorical status of statements in an article: Material for summaries is selected in such a way that summaries can highlight the new contribution of the source article and situate it with respect to earlier work. We provide a gold standard for summaries of this kind consisting of a substantial corpus of conference articles in computational linguistic annotated with human judgments of the rhetorical status and relevance of each sentence in the articles. We present several experiments measuring our judges' agreement on theses annotations. We also present an algorithm that, on the basis of the annoted training material, selects content from unseen articles and classifies it into a fixed set of seven rhetorical categories. The output of this extraction and classification system can be viewed as a single-document summary in its own right; alternatively, it provides starting material for the generation of task-oriented and user-tailored summaries designed to give users an overview of a scientific field.

Summarizing scientific articles: Experiments with relevance and rhetorical status : Summarization

The study of computerized correction of spelling errors has a relatively long history and remains of considerable current interest if regularly appearing papers on the topic are any gauge. Whereas early papers focused on the correction of output from optical character recognition (OCR), voice recognition, Morse code, or on spelling errors in program code, the application of most interest today is probably the correction of machine-readable text. Also, the techniques involved in spelling error correction have other important applications, for example, measuring the similarity of two strings of symbols to determine the evolutionary distance of proteins. The specialized subtopics are covered extensively in recent bibliographies by Peterson [7] and Pollock [8] and are not discussed further here. The most suitable correction strategy for text often depends on both its nature and its source. Correcting source code for a procedural language with a smafl vocabulary of short words (e.g., a typical programming language or the command language for a bibliographic search system) presents quite different problems than scientific text. Similarly, OCR output contains almost exclusively substitution errors, which ordinarily account for less than 20 percent of keyboarded misspellings [9, 10]. This paper describes the correction pro.gram developed as part of SPEEDCOP (Spelling Error. Detection/ ©1984ACM0001-0782/84/0400-0358 75¢ Correction Project), a Chemical Abstracts Service (CAS) project supported by the National Science Foundation. The program is intended not as a theoretical construct but as a useful tool for text editing. Under SPEEDCOP, approximately 25,000,000 words from seven scientific and scholarly textual databases were processed to extract over 50,000 misspellings using a dictionary equivalent to 40,000 words. This contrasts sharply with most work on spelling correction that tends to feature small dictionaries and either few or artificial misspellings. In our case, the use of real data gave credibility to the proposed solution, whereas the large dictionary revealed problems such as ambiguity that would not otherwise have been discovered. Internal reports [10-13] and papers [9, 19] describe in considerable detail how the misspellings were gathered and analyzed. The core of the SPEEDCOP program is an algorithm for correcting only isolated misspellings that contain a single error and whose correct forms are in a dictionary. This is not as drastic a restriction as it may seem as 90-95 percent of misspellings in raw keyboarding typically contain only one error [9, 10]. The SPEEDCOP program also incorporates a common misspelling dictionary and a function word routine and is therefore not rigidly restricted to the above class of misspellings. In practice, the program corrected 85-95 percent of the misspellings for which it was designed, 75-90 percent of those whose corresponding words were in the dic-

Automatic spelling correction in scientific and scholarly text

Work performed under the SPElling Error Detection COrrection Project (SPEEDCOP) supported by National Science Foundation (NSF) at Chemical Abstracts Service (CAS) to devise effective automatic methods of detecting and correcting misspellings in scholarly and scientific text is described. The investigation was applied to 50,000 word/misspelling pairs collected from six datasets (Chemical Industry Notes (CIN), Biological Abstracts (BA). Chemical Abstracts (CA), Americal Chemical Society primary journal keyboarding (ACS), Information Science Abstracts (ISA), and Distributed On-Line Editing (DOLE) (a CAS internal dataset especially suited to spelling error studies). The purpose of this study was to determine the utility of trigram analysis in the automatic detection and/or correction of misspellings. Computer programs were developed to collect data on trigram distribution in each dataset and to explore the potential of trigram analysis for detecting spelling errors, verifying correctly-spelled words, locating the error site within a misspelling, and distinguishing between the basic kinds of spelling errors. The results of the trigram analysis were largely independent of the dataset to which it was applied but trigram compositions varied with the dataset. The trigram analysis technique developed determined the error site within a misspelling accurately, but did not distinguish effectively between different error types or between valid words and misspellings. However, methods for increasing its accuracy are suggested.

The use of trigram analysis for spelling error detection

Automatic Abstracting Research at Chemical Abstracts Service

The SPEEDCOP (SPEIIing Error Detection correction Project) project recently completed at Chemical Abstracts Service (CAS) extracted over 50,000 misspellings from approximately 25,000,000 words of text from seven scientific and scholarly databases. The misspellings were automatically classified and the error types analyzed. The results, which were consistent over the different databases, showed that the expected incidence of misspelling is 0.2%, that 90–95% of spelling errors have only a single mistake, that substitution is homogeneous while transposition is heterogeneous, that omission is the commonest type of misspelling, and that inadvertent doubling of a letter is the most important cause of insertion errors. The more frequently a letter occurs in the text, the more likely it is to be involved in a spelling error. Most misspellings collected by SPEEDCOP are of the type colloquially referred to as “typos” and approximately 90% are unlikely to be repeated in normal spans of text.

Collection and characterization of spelling errors in scientific and scholarly text

Not only does the problem of correcting spelling errors by computer have a long history, it is evidently of considerable current interest as papers and letters on the topic continue to appear rapidly. This is not surprising, since techniques useful in detecting and correcting mis‐spellings normally have other important applications. Moreover, both the power of small computers and the routine production of machine‐readable text have increased enormously over the last decade to the point where automatic spelling error detection/correction has become not only feasible but highly desirable.

Joseph J. Pollock

Papers

Automatic spelling correction in scientific and scholarly text

The use of trigram analysis for spelling error detection

Automatic Abstracting Research at Chemical Abstracts Service

Collection and characterization of spelling errors in scientific and scholarly text

Spelling error detection and correction by computer: some notes and a bibliography