An apparatus and method are disclosed for converting an input data character stream into a variable length encoded data stream in a data compression system. The data compression system includes a history array. The history array has a plurality of entries and each entry of the history array is for storing a portion of the input data stream. The method for converting the input data character stream includes the following steps. Performing a search in a history array for the longest data string which matches the input data string. If the matching data string is found within the history buffer, the next step includes encoding the longest matching data string found by appending to the encoded data stream a tag indicating the longest matching data string was found and a string substitution code. If the matching data string is not found within the history array, the next step includes encoding the first character of the input data string by appending to the encoded data stream a raw data tag indicating that no matching data string was found and the first character of the input data string.

Data compression apparatus and method

The best schemes for text compression use large models to help them predict which characters will come next. The actual next characters are coded with respect to the prediction, resulting in compression of information. Models are best formed adaptively, based on the text seen so far. This paper surveys successful strategies for adaptive modeling that are suitable for use in practical text compression systems.The strategies fall into three main classes: finite-context modeling, in which the last few characters are used to condition the probability distribution for the next one; finite-state modeling, in which the distribution is conditioned by the current state (and which subsumes finite-context modeling as an important special case); and dictionary modeling, in which strings of characters are replaced by pointers into an evolving dictionary. A comparison of different methods on the same sample texts is included, along with an analysis of future research directions.

Modeling for text compression

An apparatus and method for converting an input data character stream into a variable length encoded data stream in a data compression system. A sliding window data compression algorithm is combined with Huffman encoding on the strings and raw bytes. The Huffman table, in a compressed form, is prepended to the encoded output data. The Huffman codes representing the shortest strings encode both the string length and part of the string offset. Assigning Huffman codes to represent the combined length and offset allows the use of a smaller sliding window size without sacrificing compression ratio. The smaller window size allows implementations in software and hardware to minimize memory usage, thus reducing cost.

Data compression apparatus and method using matching string searching and Huffman encoding

Language acquisition, data compression and generalization

The primary goal of this article is to survey state-of-the-art indexing methods for approximate dictionary searching. To improve understanding of the field, we introduce a taxonomy that classifies all methods into direct methods and sequence-based filtering methods. We focus on infrequently updated dictionaries, which are used primarily for retrieval. Therefore, we consider indices that are optimized for retrieval rather than for update. The indices are assumed to be associative, that is, capable of storing and retrieving auxiliary information, such as string identifiers. All solutions are lossless and guarantee retrieval of strings within a specified edit distance k. Benchmark results are presented for the practically important cases of k=1, 2, and 3. We concentrate on natural language datasets, which include synthetic English and Russian dictionaries, as well as dictionaries of frequent words extracted from the ClueWeb09 collection. In addition, we carry out experiments with dictionaries containing DNA sequences. The article is concluded with a discussion of benchmark results and directions for future research.

/pdf/indexing-methods-for-approximate-dictionary-searching-prq8iw22vp.pdf

Ernst J. Schuegraf

Papers

A comparison of algorithms for data base compression by use of fragments as language elements

Selection of equifrequent word fragments for information retrieval

Signature searching: theory and applications