A secure data interchange system enables information about bilateral and multilateral interactions between multiple persistent parties to be exchanged and leveraged within an environment that uses a combination of techniques to control access to information, release of information, and matching of information back to parties. Access to data records can be controlled using an associated price rule. A data owner can specify a price for different types and amounts of information access.

Secure data interchange

The recently developed technique of arithmetic coding, in conjunction with a Markov model of the source, is a powerful method of data compression in situations where a linear treatment is inappropriate. Adaptive coding allows the model to be constructed dynamically by both encoder and decoder during the course of the transmission, and has been shown to incur a smaller coding overhead than explicit transmission of the model's statistics. But there is a basic conflict between the desire to use high-order Markov models and the need to have them formed quickly as the initial part of the message is sent. This paper describes how the conflict can be resolved with partial string matching, and reports experimental results which show that mixed-case English text can be coded in as little as 2.2 bits/ character with no prior knowledge of the source.

Data Compression Using Adaptive Coding and Partial String Matching

This paper consists of an overview on universal prediction from an information-theoretic perspective. Special attention is given to the notion of probability assignment under the self-information loss function, which is directly related to the theory of universal data compression. Both the probabilistic setting and the deterministic setting of the universal prediction problem are described with emphasis on the analogy and the differences between results in the two settings.

Universal prediction

An apparatus and method are disclosed for converting an input data character stream into a variable length encoded data stream in a data compression system. The data compression system includes a history array. The history array has a plurality of entries and each entry of the history array is for storing a portion of the input data stream. The method for converting the input data character stream includes the following steps. Performing a search in a history array for the longest data string which matches the input data string. If the matching data string is found within the history buffer, the next step includes encoding the longest matching data string found by appending to the encoded data stream a tag indicating the longest matching data string was found and a string substitution code. If the matching data string is not found within the history array, the next step includes encoding the first character of the input data string by appending to the encoded data stream a raw data tag indicating that no matching data string was found and the first character of the input data string.

Data compression apparatus and method

This paper is concerned with algorithms for prediction of discrete sequences over a finite alphabet, using variable order Markov models. The class of such algorithms is large and in principle includes any lossless compression algorithm. We focus on six prominent prediction algorithms, including Context Tree Weighting (CTW), Prediction by Partial Match (PPM) and Probabilistic Suffix Trees (PSTs). We discuss the properties of these algorithms and compare their performance using real life sequences from three domains: proteins, English text and music pieces. The comparison is made with respect to prediction quality as measured by the average log-loss. We also compare classification algorithms based on these predictors with respect to a number of large protein classification tasks. Our results indicate that a "decomposed" CTW (a variant of the CTW algorithm) and PPM outperform all other algorithms in sequence prediction tasks. Somewhat surprisingly, a different algorithm, which is a modification of the Lempel-Ziv compression algorithm, significantly outperforms all algorithms on the protein classification problems.

/pdf/on-prediction-using-variable-order-markov-models-10z5ol2gkv.pdf

On prediction using variable order Markov models

The Ziv-Lempel compression algorithm is a string matching and parsing approach to data compression. The symbolwise equivalent for parsing models has been defined by Rissanen and Langdon and gives the same ideal codelength at the same cost in coding parameters. By describing the context and coding parameter for each symbol an insight is provided into how the Ziv-Lempel method achieves compression. This treatment does not employ a probabilistic source for the data string. The Ziv-Lempel method effectively counts symbol instances within parsed phrases. The coding parameter for each symbolwise context is determined by cumulative count ratios. The code string length increase for a symbol y following substring s , under the symbolwise equivalent, is the log of the ratio of node counts in subtrees s and s\cdot y of the Ziv-Lempel parsing tree. To demonstrate the symbolwise equivalent of the Ziv-Lempel algorithm, we extend the work of Rissanen and Langdon to incomplete parse trees. The result requires the proper handling of the comma when one phrase is the prefix of another phrase.

A note on the Ziv - Lempel model for compressing individual sequences (Corresp.)

This invention relates to method and apparatus for compressing multilevel signals. The compression is based upon prediction errors and probability distributions. Compression is improved by conditioning the probability distributions using context of previous events. Storage required for storing the probability distributions is reduced by partitioning the prediction errors into predetermined ranges which become the coding events and contexts. Compression is made efficient with a partition which is a function of the sign and the number of significant bits in the prediction errors.

Compression of multilevel signals

An apparatus for ensuring continuous flow through a pipeline processor as it relates to the serial decoding of FIFO Rissanen/Langdon arithmetic string code of binary sources. The pipeline decoder includes a processor (11, 23) and a finite state machine - FSM - (21) in interactive signal relation. The processor generates output binary source signals (18), status signals (31) and K component/K candidate next integer-valued control parameters (LO, k0; L1, k1). These signals and parameters are generated in response to the concurrent application of one bit from successive arithmetic code bits, a K component present integer-value control parameter (52) and K component vector representation (T, TA) of the present internal state (51) of the associated finite state machine. The FSM makes a K-way selection from K candidate next internal states and K candidate next control parameters. This selection uses no more than K 2 + K computations. The selected signals are then applied to the processor in a predetermined displaced time relation to the present signals in the processor. As a consequence, this system takes advantage of the multi-state or "memory" capability of the FSM in order to control the inter-symbol influence and facilitate synchronous multi-stage pipeline decoding.

A recursively operating information handling system and apparatus for recursively decoding an instantaneous FIFO binary arithmetic number string

Comments on "A universal data compression system" by J. Rissanen

An apparatus for ensuring continuous flow through a pipeline processor as it relates to the serial decoding of FIFO Rissanen/Langdon arithmetic string code of binary sources. The pipeline decoder includes a processor (11, 23) and a finite state machine (21, FSM) in interactive signal relation. The processor generates output binary source signals (18), status signals (WASMPS, 31) and K component/K candidate next integer-valued control parameters (L0, k0; L1, k1; 25). These signals and parameters are generated in response to the concurrent application of one bit from successive arithmetic code bits, a K component present integer-value control parameter (52) and K component vector representation (T, TA) of the present internal state (51) of the associated finite state machine (FSM). The FSM makes a K-way selection from K candidate next internal states and K candidate next control parameters. This selection uses no more than K2 +K computations. The selected signals are then applied to the processor in a predetermined displaced time relation to the present signals in the processor. As a consequence, this system takes advantage of the multi-state or "memory" capability of an FSM in order to control the inter-symbol influence and facilitate synchronous multi-stage pipeline decoding.

Method and means for pipeline decoding of the high to low order pairwise combined digits of a decodable set of relatively shifted finite number of strings

Jr. Glen George Langdon

Papers

A note on the Ziv - Lempel model for compressing individual sequences (Corresp.)

Compression of multilevel signals

A recursively operating information handling system and apparatus for recursively decoding an instantaneous FIFO binary arithmetic number string

Comments on "A universal data compression system" by J. Rissanen

Method and means for pipeline decoding of the high to low order pairwise combined digits of a decodable set of relatively shifted finite number of strings