SUMMARY We show that suffix trees store various kinds of redundant information. We exploit these redundancies to obtain more space efficient representations. The most space efficient of our representations requires 20 bytes per input character in the worst case, and 10.1 bytes per input character on average for a collection of 42 files of different type. This is an advantage of more than 8 bytes per input character over previous work. Our representations can be constructed without extra space, and as fast as previous representations. The asymptotic running times of suffix tree applications are retained. Copyright © 1999 John Wiley & Sons, Ltd.

Reducing the space requirement of suffix trees

This article describes the dynamic prefix tries, a novel data structure with algorithms for insertion, deletion, and retrieval to build and maintain a dynamic database of binary keys of arbitrary length. These tries extend the concepts of compact digital (Patricia) tries to support the storage of prefixes and to guarantee retrieval times at most linear in the length of the input key irrespective of the trie size, even when searching for longest-matching prefixes. The new design permits very efficient, simple and nonrecursive implementations of small code size and minimal storage requirements. Insert and delete operations have strictly local effects, and their particular sequence is irrelevant for the structure of the resulting trie, thus maintaining at all times the desired storage and computational efficiency. The algorithms have bees successfully employed in experimental communication systems and products for a variety of networking functions such as address resolution, maintenance and verification of access control lists, and high-performance routing tables in operating system kernels.

Routing on longest-matching prefixes

As the broadband access technologies, such as DSL, cable modem, and gigabit Ethernet, are providing affordable broadband solutions to the Internet from home and the enterprise, it is required to build next generation routers with high-speed interfaces (e.g., 10 or 40 Gb/s) and large switching capacity (e.g., multipetabit). This paper first points out the issues of building such routers, such as memory speed constraint, packet arbitration bottleneck, and interconnection complexity. It then presents several algorithms/architectures to implement IP route lookup, packet classification, and switch fabrics. Some of the functions, such as packet classification, route lookup, and traffic management, can be implemented with emerging network processors that have the advantages of providing flexibility to new applications and protocols, shortening the design cycle and time-to-market, and reducing the implementation cost by avoiding the ASIC approach. Several proposed algorithms for IP route lookup and packet classification are compared in respect to their search/update speeds and storage requirements. Different efficient arbitration schemes for output port contention resolution are presented and analyzed. The paper also surveys various switch architectures of commercial routers and switch chip sets. At the end, it outlines several challenging issues that remain to be researched for next generation routers.

Next generation routers

Many applications depend on efficient management of large sets of distinct strings in memory. For example, during index construction for text databases a record is held for each distinct word in the text, containing the word itself and information such as counters. We propose a new data structure, the burst trie, that has significant advantages over existing options for such applications: it uses about the same memory as a binary search tree; it is as fast as a trie; and, while not as fast as a hash table, a burst trie maintains the strings in sorted or near-sorted order. In this paper we describe burst tries and explore the parameters that govern their performance. We experimentally determine good choices of parameters, and compare burst tries to other structures used for the same task, with a variety of data sets. These experiments show that the burst trie is particularly effective for the skewed frequency distributions common in text collections, and dramatically outperforms all other data structures for the task of managing strings while maintaining sort order.

Burst tries: a fast, efficient data structure for string keys

We present an efficient implementation of a write-only top-down construction for suffix trees. Our implementation is based on a new, space-efficient representation of suffix trees which requires only 12 bytes per input character in the worst case, and 8:5 bytes per input character on average for a collection of files of different type. We show how to efficiently implement the lazy evaluation of suffix trees such that a subtree is evaluated not before it is traversed for the first time. Our experiments show that for the problem of searching many exact patterns in a fixed input string, the lazy top-down construction is often faster and more space efficient than other methods.

/pdf/efficient-implementation-of-lazy-suffix-trees-4j97m71z5w.pdf

Efficient Implementation of Lazy Suffix Trees

SUMMARY A new internal array structure, called a double-array, implementing a trie structure is presented. The double-array combines the fast access of a matrix form with the compactness of a list form. The algorithms for retrieval, insertion and deletion are introduced through examples. Although insertion is rather slow, it is still practical, and both the deletion and the retrieval time can be improved from the list form. From the comparison with the list for various large sets of keys, it is shown that the size of the double-array can be about 17 per cent smaller than that of the list, and that the retrieval speed of the double-array can be from 3·1 to 5·1 times faster than that of the list.

An efficient implementation of trie structures

A trie structure is frequently used for various applications, such as natural language dictionaries, database systems and compilers. However, the total number of states of a trie (and transitions between them) becomes large, so that the space cost may not be acceptable for a huge key set. In order to resolve this disadvantage, this paper presents a new scheme, called a "two-trie", that enables us to perform efficient retrievals, insertions and deletions for the key sets. The essential idea is to construct two tries for both front and rear compressions of keys, which is similar to a DAWG (directed acyclic word-graph). The approach differs from a DAWG in that the two-trie approach presented can uniquely determine information corresponding to keys while a DAWG cannot. For an efficient implementation of the two-trie, two types of data structures are introduced. Theoretical and experimental observations show that the method presented is more practical than existing ones considering the use of dynamic key sets, information storage of keys and compression of transitions.

A trie compaction algorithm for a large set of keys

A trie structure can immediately determine whether a desired key is in a given key set or not, and can find its longest match easily. Thanks to these attractive properties, a trie structure is frequently used for various fields, such as natural language dictionaries, database systems and compilers. However, the total number of states of a trie becomes large, so space requirements are not good for a huge key set. To resolve this disadvantage a new structure which reduces the total number of states in a traditional trie, called a double-trie, is introduced in this paper. Insertion and deletion operations, as well as key retrieval for this double-trie, are presented. The efficiency of this method is shown by the results of simulations for various key sets.

A method of compressing trie structures

SCOPE: Algorithms, Information storage and retrieval. A trie is a search tree obtained by merging the common suffixes of the key set. It has the advantage that all keys as prefixes of an input string can be retrieved with high speed. When the size of the key set is enlarged, however, a problem arises, as the number of transitions increases, so too does the need for a large storage capacity. This paper proposes an algorithm that dynamically constructs DAWGs (Directed Acyclic Word Graphs) for the handling of dynamic key sets. It also solves the problem of the increasing number of transitions in the trie structure. The proposed method constructs a DAWG through the local separation of common suffixes for updating a key and, after finishing updating a key, the local transition merge of common suffixes. The proposed algorithm is theoretically evaluated and the data structure for the implementation is discussed. Experimental results show that the number of transitions in the DAWG is reduced by approx. 50 to 70% ...

An algorithm for dynamic processing of dawg's

The trie has the feature that the retrieval can be executed with the character symbols composing the key as the unit, and a high-speed retrieval is realized independently of the total number of keys. Consequently, it used frequently in the search of the natural language dictionary and in other problems. A problem, however, is that the number of trie states increases with the enlargement of the key set, which necessitates a larger memory capacity. To remedy this point, DAWG (Directed Acyclic Word-Graph) is proposed, where the common suffix of the tries is compressed.



Then, a new problem arises in that the record information cannot be determined uniquely for the key. For this problem, this paper introduces a new structure, where the number of states is reduced by merging the common suffixes of the tries, while determining uniquely the record information for the key. The algorithm for retrieval, insertion and deletion of the key is proposed for the structure. In the proposed method, the set of keys is represented using two tries. One of the tries memorizes the prefix of the minimum length that can discriminate uniquely between the key from other keys. The other trie stores the suffixes of the remaining keys in order to merge the common suffix. A simulation is executed for various sets of keys such as Chinese character called Kanji, alphabets and Japanese Katakana characters, and it is seen that the number of states is reduced by approximately 30 to 65 percent for the key set of 50,000 words, compared to the ordinary trie.

Katsushi Morimoto

Papers

An efficient implementation of trie structures

A trie compaction algorithm for a large set of keys

A method of compressing trie structures

An algorithm for dynamic processing of dawg's

A dictionary retrieval algorithm using two trie structures