An apparatus and method are disclosed for converting an input data character stream into a variable length encoded data stream in a data compression system. The data compression system includes a history array. The history array has a plurality of entries and each entry of the history array is for storing a portion of the input data stream. The method for converting the input data character stream includes the following steps. Performing a search in a history array for the longest data string which matches the input data string. If the matching data string is found within the history buffer, the next step includes encoding the longest matching data string found by appending to the encoded data stream a tag indicating the longest matching data string was found and a string substitution code. If the matching data string is not found within the history array, the next step includes encoding the first character of the input data string by appending to the encoded data stream a raw data tag indicating that no matching data string was found and the first character of the input data string.

Data compression apparatus and method

The best schemes for text compression use large models to help them predict which characters will come next. The actual next characters are coded with respect to the prediction, resulting in compression of information. Models are best formed adaptively, based on the text seen so far. This paper surveys successful strategies for adaptive modeling that are suitable for use in practical text compression systems.The strategies fall into three main classes: finite-context modeling, in which the last few characters are used to condition the probability distribution for the next one; finite-state modeling, in which the distribution is conditioned by the current state (and which subsumes finite-context modeling as an important special case); and dictionary modeling, in which strings of characters are replaced by pointers into an evolving dictionary. A comparison of different methods on the same sample texts is included, along with an analysis of future research directions.

Modeling for text compression

As a result of the rapid growth of the volume of electronic data, text compression and indexing techniques are receiving more and more attention. These two issues are usually treated as independent problems, but approaches of combining them have recently attracted the attention of researchers. In this thesis, we review and test some of the more effective and some of the more theoretically interesting techniques. Various compression and indexing techniques are presented, and we also present two compressed text indices. Based on these techniques, we implement an compressed full-text index, so that compressed texts can be indexed to support fast queries without decompressing the whole texts. The experiments show that our index is compact and supports fast search.

Indexing Compressed Text

Several methods are presented for adaptive, invertible data compression in the style of Lempel's and Ziv's first textual substitution proposal. For the first two methods, the article describes modifications of McCreight's suffix tree data structure that support cyclic maintenance of a window on the most recent source characters. A percolating update is used to keep node positions within the window, and the updating process is shown to have constant amortized cost. Other methods explore the tradeoffs between compression time, expansion time, data structure size, and amount of compression achieved. The article includes a graph-theoretic analysis of the compression penalty incurred by our codeword selection policy in comparison with an optimal policy, and it includes empirical studies of the performance of various adaptive compressors from the literature.

/pdf/data-compression-with-finite-windows-3mva2vnyfn.pdf

Data compression with finite windows

A programmable logic device (PLD) can be configured using configuration data stored on a memory. The configuration data is compressed using a compression algorithm before being stored on the memory. When the PLD is to be configured, the compressed configuration data is read from the memory, decompressed, then loaded onto the PLD.

Configuring a programmable logic device

A new technique for compression of character strings is presented. The technique is based on the use of a dictionary forest which is built simultaneously with the encoding and decoding. Codes representing substrings are addresses in the dictionary forest. Experimental results show that the length of the text can be reduced more than 50% with no a priori knowledge of the nature of the text.

Compression of character strings by an adaptive dictionary

Evaluation of a hierarchical bit-vector compression technique

Le but de la technique de compression de textes appliquee aux fichiers informatises est de reduire l'espace necessaire aux fichiers ce qui permet soit de stocker plus de donnees soit d'economiser sur les couts de transmission des donnees. Une methode pour une compression universelle de textes est presentee et analysee

One‐pass text compression with a subword dictionary

/pdf/sampling-without-replacement-in-linear-time-1fr8uejczj.pdf

Matti Jakobsson

Papers

Compression of character strings by an adaptive dictionary

Evaluation of a hierarchical bit-vector compression technique

One‐pass text compression with a subword dictionary

Sampling without Replacement in Linear Time