We survey the current techniques to cope with the problem of string matching that allows errors. This is becoming a more and more relevant issue for many fast growing areas such as information retrieval and computational biology. We focus on online searching and mostly on edit distance, explaining the problem and its relevance, its statistical behavior, its history and current developments, and the central ideas of the algorithms and their complexities. We present a number of experiments to compare the performance of the different algorithms and show which are the best choices. We conclude with some directions for future work and open problems.

/pdf/a-guided-tour-to-approximate-string-matching-22fowlmq5b.pdf

A guided tour to approximate string matching

String data is ubiquitous, and its management has taken on particular importance in the past few years. Approximate queries are very important on string data especially for more complex queries involving joins. This is due, for example, to the prevalence of typographical errors in data, and multiple conventions for recording attributes such as name and address. Commercial databases do not support approximate string joins directly, and it is a challenge to implement this functionality efficiently with user-defined functions (UDFs). In this paper, we develop a technique for building approximate string join capabilities on top of commercial databases by exploiting facilities already available in them. At the core, our technique relies on matching short substrings of length , called -grams, and taking into account both positions of individual matches and the total number of such matches. Our approach applies to both approximate full string matching and approximate substring matching, with a variety of possible edit distance functions. The approximate string match predicate, with a suitable edit distance threshold, can be mapped into a vanilla relational expression and optimized by conventional relational optimizers. We demonstrate experimentally the benefits of our technique over the direct use of UDFs, using commercial database systems and real data. To study the I/O and CPU behavior of approximate string join algorithms with variations in edit distance and -gram length, we also describe detailed experiments based on a prototype implementation.

/pdf/approximate-string-joins-in-a-database-almost-for-free-3cak24s904.pdf

Approximate String Joins in a Database (Almost) for Free

/pdf/approximate-string-joins-in-a-database-almost-for-free-3fxp1nstzl.pdf

Approximate String Joins in a Database (Almost) for Free -- Erratum

The approximate string matching problem is to find all locations at which a query of lengthm matches a substring of a text of length n with k-or-fewer differences. Simple and practical bit-vector algorithms have been designed for this problem, most notably the one used in agrep. These algorithms compute a bit representation of the current state-set of the k-difference automaton for the query, and asymptotically run in either O(nm/w) or O(nm log s/w) time where w is the word size of the machine (e.g., 32 or 64 in practice), and s is the size of the pattern alphabet. Here we present an algorithm of comparable simplicity that requires only O(nm/w) time by virtue of computing a bit representation of the relocatable dynamic programming matrix for the problem. Thus, the algorithm's performance is independent of k, and it is found to be more efficient than the previous results for many choices of k and smallm.Moreover, because the algorithm is not dependent on k, it can be used to rapidly compute blocks of the dynamic programming matrix as in the 4-Russians algorithm of Wu et al.(1996). This gives rise to an O(kn/w) expected-time algorithm for the case where m may be arbitrarily large. In practice this new algorithm, that computes a region of the dynamic progr amming (d.p.) matrx w entries at a time using the basic algorithm as a subroutine is significantly faster than our previous 4-Russians algorithm, that computes the same region 4 or 5 entries at a time using table lookup. This performance improvement yields a code that is either superior or competitive with all existing algorithms except for some filtration algorithms that are superior when k/m is sufficiently small.

/pdf/a-fast-bit-vector-algorithm-for-approximate-string-matching-qcfnwyk3dt.pdf

A fast bit-vector algorithm for approximate string matching based on dynamic programming

We present in this paper two algorithms. The first one extracts repeated motifs from a sequence defined over an alphabet Σ. For instance, Σ may be equal to (A, C, G, T} and the sequence represents an encoding of a DNA macromolecule. The motifs searched correspond to words over the same alphabet which occur a minimum number q of times in the sequence with at most e mismatches each time (q is called the quorum constraint). The second algorithm extracts common motifs from a set of N ≥ 2 sequences. In this case, the motifs must occur, again with at most e mismatches, in 1 ≤ q ≤ N distinct sequences of the set. In both cases, the words representing the motifs may never be present exactly in the sequences. We therefore speak of the motifs, repeated in a sequence or common to a set of them, as being external objects and denote them by the expression valid models if they verify the quorum constraint q. The approach we introduce here for finding all valid models corresponding to either repeated or common motifs starts by building a suffix tree of the sequence(s) and then, after some further preprocessing, uses this tree to simply spell the models. Assuming an alphabet of fixed size, the total time needed is O(nN 2 V(e, κ)) using O(nN 2 /ω) space, where n is the (average) length of the sequence(s), k is the length of the models sought or is the length of the longest possible valid models, ω is the size of a word machine and V(e,κ) is the number of words of length κ at a Hamming distance at most e from another κ-length word. V(e,κ) may be majored by k e |Σ| e . This improves on an algorithm by Waterman [23]. It is also a better time bound than our previous approach [15] for the common motifs problem whenever N < k|Σ|, and a better space bound when N/ω < k. It is a better time and space bound in absolute for the repeated motifs problem. The complexities obtained in this second case are O(nV(e,k)) and O(n) respectively. Finally, we suggest how to extend these algorithms to deal with gaps.

Spelling Approximate Repeated or Common Motifs Using a Suffix Tree

Let T be a text of length n and P a pattern of length m, both strings over a fixed finite alphabet σ. We wish to find all approximate occurrences of P in T having weighted edit distance at most k from P: this is the approximate substring matching problem. We focus on the case in which T is fixed and preprocessed in linear time, while P and k vary over consecutive searches. We give an O(mq+t vanocc ) time and O(q) space algorithm, where q≤n depends on the problem instance, and t vanocc is the size of the output. The running time is proportional to the amount of matching, in the worst case as fast as standard dynamic programming. The algorithm uses the suffix tree representation of the text. The best previous algorithm requires O(mq log q+t vanocc ) time and O(mq) space.

Fast approximate matching using suffix trees

Let two strings S, T over a finite alphabet Σ be given, and let M be an arbitrary relation on Σ×Σ. Define an approximate match (x,y) of two length m subwords (substrings) x ⊑ S, y ⊑ T when M(x i ,y i , for all 1≤i≤m. A match implies all the local alignments (without insertions and deletions) which are pairings of specific occurrances of x and y. A match (x,y) is maximal if there exists no longer match (u, v) such that all of the local alignments implied by (x,y) are contained in a local alignment implied by (u,v). We give an efficient algorithm for finding all maximal matches between S and T. The algorithm runs in time bounded by the sum of the lengths of the maximal matches, at worst. O(¦Σ¦2n2). The main application is identifying homologous regions of protein sequences.

Fast Identification of Approximately Matching Substrings

A few years ago, my team was working on a commercial Java development project for Enhanced 911 (E911) emergency call centers. We were frustrated by trying to meet the data-storage requirements of this project using the traditional model of Java over an SQL database. After some reflection about the particular requirements (and nonrequirements) of the project, we took a deep breath and decided to create our own custom persistence layer from scratch.

Archie L. Cobbs

Papers

Fast approximate matching using suffix trees

Fast Identification of Approximately Matching Substrings

Persistence Programming