Logical information theory: new logical foundations for information theory

TL;DR: The definition of Shannon entropy as well as the notions on joint, conditional, and mutual entropy as defined by Shannon can all be derived by a uniform transformation from the corresponding formulas of logical information theory, which provides the set- theoretic and measure-theoretic foundations for information theory.

Abstract: here is a new theory of information based on logic. The definition of Shannon entropy as well as the notions on joint, conditional, and mutual entropy as defined by Shannon can all be derived by a uniform transformation from the corresponding formulas of logical information theory. Information is first defined in terms of sets of distinctions without using any probability measure. When a probability measure is introduced, the logical entropies are simply the values of the (product) probability measure on the sets of distinctions. The compound notions of joint, conditional, and mutual entropies are obtained as the values of the measure, respectively, on the union, difference, and intersection of the sets of distinctions. These compound notions of logical entropy satisfy the usual Venn diagram relationships (e.g., inclusion-exclusion formulas) since they are values of a measure (in the sense of measure theory). The uniform transformation into the formulas for Shannon entropy is linear so it explains the long-noted fact that the Shannon formulas satisfy the Venn diagram relations--as an analogy or mnemonic--since Shannon entropy is not a measure (in the sense of measure theory) on a given set. What is the logic that gives rise to logical information theory? Partitions are dual (in a category-theoretic sense) to subsets, and the logic of partitions was recently developed in a dual/parallel relationship to the Boolean logic of subsets (the latter being usually mis-specified as the special case of "propositional logic"). Boole developed logical probability theory as the normalized counting measure on subsets. Similarly the normalized counting measure on partitions is logical entropy--when the partitions are represented as the set of distinctions that is the complement to the equivalence relation for the partition. In this manner, logical information theory provides the set-theoretic and measure-theoretic foundations for information theory. The Shannon theory is then derived by the transformation that replaces the counting of distinctions with the counting of the number of binary partitions (bits) it takes, on average, to make the same distinctions by uniquely encoding the distinct elements--which is why the Shannon theory perfectly dovetails into coding and communications theory.

Summary

1 Introduction

• This paper develops the logical theory of information-as-distinctions.
• Partitions are dual (in a category-theoretic sense) to subsets.
• This paper develops the normalized counting measure on partitions as the analogous quantitative treatment in the logic of partitions.
• That question is addressed in this paper by showing that there is a transformation of formulas that transforms each of the logical entropy compound formulas into the corresponding Shannon entropy compound formula, and the transform preserves the Venn diagram relationships that automatically hold for measures.

2 Logical information as the measure of distinctions

• There is now a widespread view that information is fundamentally about differences, distinguishability, and distinctions.
• This view even has an interesting history.
• It is more convenient, indeed, that these Differences should be of as great Variety as the Letters of the Alphabet; but it is suffi cient if they be but twofold, because Two alone may, with somewhat more Labour and Time, be well enough contrived to express all the rest. [47, Chap.
• Thus counting distinctions [12] would seem the right way to measure information,1 and that is the measure which emerges naturally out of partition logic—just as finite logical probability emerges naturally as the measure of counting elements in Boole’s subset logic.
• By the very essence of this discipline, the foundations of information theory have a finite combinatorial character. [27, p. 39].

3 Duality of subsets and partitions

• Hence the authors will start with a brief review of the relationship between these two dual forms of 1This paper is about what Adriaans and van Benthem call "Information B: Probabilistic, information-theoretic, measured quantitatively", not about "Information A: knowledge, logic, what is conveyed in informative answers" where the connection to philosophy and logic is built-in from the beginning.
• Likewise, the paper is not about Kolmogorovstyle "Information C: Algorithmic, code compression, measured quantitatively." [4, p. 11] 2Kolmogorov had something else in mind such as a combinatorial development of Hartley’s log (n) on a set of n equiprobable elements.[28] mathematical logic.
• Modern category theory shows that the concept of a subset dualizes to the concept of a quotient set, equivalence relation, or partition.
• Bi that are mutually disjoint and jointly exhaustive (∪iBi = U).
• Thus the information set or infoset associated with a partition π is ditset dit (π).

4 Classical subset logic and partition logic

• The logical partition operations can also be defined in terms of the corresponding logical operations on subsets.
• An equivalence relation is reflexive, symmetric, and transitive.
• This usage is consistent with calling the subsets that equal their rst-closures closed subsets of U × U (so closed subsets = equivalence relations) so the complements are the open subsets (= partition relations).
• And then the definitions of a valid formula are also parallel, namely, no matter what is substituted for the variables, the whole formula evaluates to the top of the algebra.

5 Classical logical probability and logical entropy

• In Gian-Carlo Rota’s Fubini Lectures [38] (and in his lectures at MIT), he has remarked in view of duality between partitions and subsets that, quantitatively, the “lattice of partitions plays for information the role that the Boolean algebra of subsets plays for size or probability”[29, p. 30] or symbolically: information : partitions :: probability : subsets.
• The formulas for mutual information, joint entropy, and conditional entropy are defined so these Shannon entropies satisfy Venn diagram formulas (e.g., [1, p. 109]; [35, p. 508]) that would follow automatically if Shannon entropy were a measure in the technical sense.
• Thus all of Campbell’s desiderata are true when: “sets”= ditsets, the set of distinctions of partitions (or, in general, information sets of distinctions), and “entropies”= normalized counting measure of the ditsets (or, in general, product probability measure on the infosets), i.e., the logical entropies.
• This formalization exercise seems to have been first carried out by Guo Ding Hu [25] but was also noted by Imre Csiszar and Janos Körner [11], and redeveloped by Raymond Yeung ([48]; [49]).

7 The dit-bit transform

• The logical entropy formulas for various compound notions (e.g., conditional entropy, mutual information, and joint entropy) stand in certain Venn diagram relationships because logical entropy is a measure.
• Indeed, there is such a connection, the dit-bit transform.
• Indeed, the bit-count is a “coarser-grid”that loses some information in comparison to the dit-count as shown by the analysis of mutual information.
• The dit-bit connection carries over to all the compound notions of entropy so that the Shannon notions of conditional entropy, mutual information, cross-entropy, and divergence can all be developed from the corresponding notions for logical entropy.
• That is why the Shannon formulas can satisfy the Venn diagram relationships even though Shannon entropy is not a measure.

8 Information algebras and joint distributions

• These information sets or infosets are defined solely in terms of equations and inequations (the ‘calculus of identity and difference’) independent of any probability distributions.
• Even though all the 15 subsets in the information algebra aside from the empty set ∅ are always nonempty, some of the logical entropies can still be 0.
• Consider SX⊃Y = SX∧Y ∪ SY ∧¬X ∪ S¬X∧¬Y and suppose that the probability distribution gives µ.

9.1 Logical conditional entropy

• All the compound notions for Shannon and logical entropy could be developed using either partitions (with point probabilities) or probability distributions of random variables as the given data.
• Since the treatment of Shannon entropy is most often in terms of probability distributions, the authors will stick to that case for both types of entropy.
• The formula for the compound notion of logical entropy will be developed first, and then the formula for the corresponding Shannon compound entropy will be obtained by the dit-bit transform.
• For the definition of the conditional entropy h (X|Y ), the authors simply take the product measure of the set of pairs (x, y) and (x′, y′) that give an X-distinction but not a Y -distinction.
• This logical distance is a Hamming-style distance function [34, p. 66] based on the difference between the random variables.

9.2 Shannon conditional entropy

• All the Shannon notions can be obtained by the dit-bit transform of the corresponding logical notions.
• Since the dit-bit transform preserves sums and differences, the authors will have the same sort of Venn diagram formula for the Shannon entropies and this can be illustrated in the analogous “mnemonic” Venn diagram.

10.1 Logical mutual information

• Logical mutual information in a joint probability distribution.
• It is a non-trivial fact that two nonempty partition ditsets always intersect [12].
• The same holds for the positive supports of the basic infosets SX and SY .

11 Independent Joint Distributions

• Independence means the joint probability p (x, y) can always be separated into p (x) times p (y).
• This carries over to the standard two-draw probability interpretation of logical entropy.
• This is a striking difference between the average bit-count Shannon entropy and the dit-count logical entropy.
• Aside from the waste case where h (X)h (Y ) = 0, there are always positive probability mutual distinctions for X and Y , and that dit-count information is not recognized by the coarser-grained average bit-count Shannon entropy.

12 Cross-entropies and divergences

• The probability that the points are distinct would be a natural and more general notion of logical entropy that would be the: h (p‖q) = ∑ i pi(1− qi) =.
• Then the information inequality implies that the logical cross-entropy is greater than or equal to the average of the logical entropies, i.e., the non-negativity of the Jensen difference: h (p||q) ≥ h(p)+h(q)2 with equality iff p = q.
• Since the logical divergence d (p||q) is symmetrical, it develops via the dit-bit transform to the symmetrized version Ds (p||q) of the Kullback-Leibler divergence.
• The logical divergence d (p||q) is clearly a distance function (or metric) on probability distributions, but even the symmetrized Kullback-Leibler divergence Ds (p||q) may fail to satisfy the triangle inequality [11, p. 58] so it is not a distance metric.

14 Entropies for multivariate joint distributions

• For the logical analysis of variation of categorical data, the logical entropies in the multivariate case divide up the variation into the natural parts.
• To emphasize that Venn-like diagrams are only a mnemonic analogy, Norman Abramson gives an example [1, pp. 130-1] where the Shannon mutual information of three variables is negative.
• Indeed, as Imre Csiszar and Janos Körner remark:.
• There are three non-trivial partitions on a set with three elements and those partitions have no dits in common.
• In probability theory, for a joint probability distribution of 3 or more random variables, there is a distinction between the variables being pair-wise independent and being mutually independent.

15 Logical entropy and some related notions

• This relationship between the Shannon/von Neumann entropies and the logical entropies in the classical and quantum cases is responsible for presenting the logical entropy as a ‘linear’approximation to the Shannon or von Neumann entropies since 1 − pi is the linear term in the series for − ln (pi) [before the multiplication by pi to make the term quadratic!].
• The authors find this framework of partitions and distinction most suitable (at least conceptually) for describing the problems of quantum state discrimination, quantum cryptography and in general, for discussing quantum channel capacity.
• There are many older results derived under the misnomer “linear entropy” or derived for the quadratic special case of the Tsallis-Havrda-Charvat entropy ([24]; [44], [45]).
• In accordance with its quadratic nature, logical entropy is the logical special case of C. R. Rao’s quadratic entropy [36].

17 Concluding remarks

• The quantitative logical entropy of an information set is the value of the product probability measure on the set.
• Since conventional information theory has heretofore been focused on the original notion of Shannon entropy (and quantum information theory on the corresponding notion of von Neumann entropy), much of the paper has compared the logical entropy notions to the corresponding Shannon entropy notions.
• Logical entropy, like logical probability, is a measure, while Shannon entropy is not.
• In sum, the logical theory of information-as-distinctions is the ground level logical theory of information stated first in terms of sets of distinctions and then in terms of two-draw probability measures on the sets.
• The Shannon information theory is a higher-level theory that requantifies distinctions by counting the minimum number of binary partitions (bits) that are required, on average, to make all the same distinctions, i.e., to encode the distinguished elements—and is thus well-adapted for the theory of coding and communication.

Figures

Figure 7: Negative ‘area’I (X,Y, Z) in ‘Venn diagram.’

Figure 2: h (X,Y ) = h (X|Y ) + h (Y ) = h (Y |X) + h (X).

Table 6: The dit-bit transform from logical entropy to Shannon entropy

Table 5: Comparisons between Shannon and logical entropy formulas

Table 1: Duality between subset logic and partition logic

Table 4: Logical entropy relationships under independence.

Figure 6: Venn diagram for logical entropies.

Table 3: Truth table for atoms in the Information Algebra

Table 7: Abramson’s example giving negative Shannon mutual information I (X,Y, Z).

Table 2: Classical logical probability theory and classical logical information theory

Figure 4: h (X,Y ) = h (X|Y ) + h (Y |X) +m (X,Y )

Figure 3: H (X,Y ) = H (X|Y ) +H (Y ) = H (Y |X) +H (X).

Full text

LOGIC JOURNAL of the IGPL
Volume 22 • Issue 1 • February 2014
Contents
Original Articles
A Routley–Meyer semantics for truth-preserving and well-determined
Łukasiewicz 3-valued logics 1
Gemma Robles and José M. Méndez
Tarski’s theorem and liar-like paradoxes 24
Ming Hsiung
Verifying the bridge between simplicial topology and algebra: the
Eilenberg–Zilber algorithm 39
L. Lambán, J. Rubio, F. J. Martín-Mateos and J. L. Ruiz-Reina
Reasoning about constitutive norms in BDI agents 66
N. Criado, E. Argente, P. Noriega and V. Botti
An introduction to partition logic 94
David Ellerman
On the interrelation between systems of spheres and epistemic
entrenchment relations 126
Maurício D. L. Reis
On an inferential semantics for classical logic 147
David Makinson
First-order hybrid logic: introduction and survey 155
Torben Braüner
Applications of ultraproducts: from compactness to fuzzy
elementary classes 166
Pilar Dellunde
LOGIC JOURNAL
of the
IGPL
Volume 22 • Issue 1 • February 2014
www.oup.co.uk/igpl
LOGIC JOURNAL
of the
IGPL
EDITORS-IN-CHIEF
A. Amir
D. M. Gabbay
G. Gottlob
R. de Queiroz
J. Siekmann
EXECUTIVE EDITORS
M. J. Gabbay
O. Rodrigues
J. Spurr
INTEREST GROUP IN PURE AND APPLIED LOGICS
ISSN PRINT 1367-0751
ISSN ONLINE 1368-9894
JIGPAL-22(1)Cover.indd 1JIGPAL-22(1)Cover.indd 1 16-01-2014 19:02:0516-01-2014 19:02:05

Logical Information Theory:
New Logical Foundations for Information Theory
[Forthcoming in: Logic Journal of the IGPL]
David Ellerman
Philosophy Department,
University of California at Riverside
June 7, 2017
Abstract
There is a new theory of infor mation based on logic. The de…n ition of Shannon entropy as
well as the no tions on joint, conditional, and mutual entropy as de…ned by Shan non can al l
be derived by a uniform transformation fr om the corresponding formulas of logical informati on
theory. Information is …rs t de…ned in terms of sets of distinctions without using any probability
measure. When a probability measure is introduced, the logical entropie s are simply the values
of the (product) probability measu re on t he se ts of distinctions. The c ompound notions of joint,
conditional, and mutual entropies are ob tained as the values of the measure, respectively, on th e
union, di¤erence, and intersection of the sets of distinctions. These compound notions of logical
entropy satisfy the usual Venn diagram relationships (e.g., inclusion-exclusion formulas) since
they are values of a measure (in the s ense of meas ure theory). The u niform transformation into
the f ormulas for Shannon entropy is linear so it expl ains the long-noted fact that t he Shannon
formulas satisfy the Venn d iagram relations–as an a nalogy or mnemonic–since Shannon entropy
is not a measure (in the sense of measure theory) on a given set.
W hat is the logic that gives rise to l ogical info rmation theory? Partitions are dual (in a
category- theoretic sense) to subsets, and the logic of partitions was recently developed in a
dual/parallel relationship to the Boolean logic of subsets (the latter being u sually mis-speci…ed
as the special case of “propositional lo gic”). Boole developed logical probability theory as the
normalized counting measure o n subsets. Similarly the normalized counting measure on parti-
tions is logical entropy–when the partitions are represented as the set of distinctions that is the
complement to the equivalence relation f or the partition.
In this manner, logic al informa tion theory provides the set-theoretic and measure-theoretic
foundations for information theory. The Shannon theory is then derived by the transformation
that r eplaces the counting o f distinctions wi th the counting of the number of binary partitions
(bits) it takes, on average, to make the same distinctions by uniquely encodi ng the distinct
elements–which is why the Shannon theory perfectly dovetails into coding and com munications
theory.
Key words: parti tion logic, logical entropy, Shannon entropy
Contents
1 Introduction 2
2 Logical information as the measure of distinctions 3
3 Duality of subsets and partitions 4
1

4 Classical subset logic and partition logic 5
5 Classical logical probability and logical entropy 6
6 Entropy as a measure of information 8
7 The dit-bit transform 10
8 Information algebras and joint distributions 11
9 Conditional entropies 13
9.1 Logical conditional entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
9.2 Shannon conditional entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
10 Mutual information 15
10.1 Logical mutual information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
10.2 Shannon mutual information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
11 Independent Joint Distributions 17
12 Cross-entropies and divergences 18
13 Summary of formulas and dit-bit transforms 20
14 Entropies for multivariate joint distributions 20
15 Logical entropy and some related notions 24
16 The statistical interpretation of Shannon entropy 25
17 Concluding remarks 28
1 Introduction
This paper develops the logical theory of information-as-distinctions. It can be seen as the application
of the logic of partitions [15] to information theory. Partitions are dual (in a category-theoretic sense)
to subsets. George Boole developed the notion of logical probability [7] as the normalized counting
measure on subsets in his logic of subsets. This paper develops the normalized counting measure on
partitions as the analogous quantitative treatment in the logic of partitions. The resulting measure is
a new logical derivation of an old formula measuring diversity and distinctions, e.g., Corrado Gini’s
index of mutability or diversity [19], that goes back to the early 20th century. In view of the idea
of information as being based on distinctions (see next section), I refer to this logical measure of
distinctions as "logical entropy".
This raises the question of the relationship of logical entropy to Claude Shannon’s entropy
([40]; [41]). The entropies are closely related since they are both ultimately based on the concept
of information-as-distinctions–but they represent two di¤erent way to quantify distinctions. Logical
entropy directly counts the distinctions (as de…ned in partition logic) whereas Shannon entropy, in
e¤ect, counts the minimum number of binary partitions (or yes/no questions) it takes, on average,
to uniquely determine or designate the distinct entities. Since that gives (in standard examples) a
binary code for the distinct entities, the Shannon theory is perfectly adapted for applications to the
theory of coding and communications.
2

The logical theory and the Shannon theory are also related in their compound notions of joint
entropy, conditional entropy, and mutual information. Logical entropy is a measure in the math-
ematical sense, so as with any measure, the compound formulas satisfy the usual Venn-diagram
relationships. The compound notions of Shannon entropy were de…ned so that they also satisfy
similar Venn diagram relationships. However, as various information theorists, principally Lorne
Campbell, have noted [9], Shannon entropy is not a measure (outside of the standard example of 2
n
equiprobable distinct entities where it is the count n of the number of yes/no questions necessary to
unique determine or encode the distinct entities)–so one can conclude only that the "analogies pro-
vide a convenient mnemonic" [9, p. 112] in terms of the usual Venn diagrams for measures. Campbell
wondered if there might be a "deeper foundation" [9, p. 112] to clarify how the Shannon formulas
can be de…ned to satisfy the measure-like relations in spite of not being a measure. That question is
addressed in this paper by showing that there is a transformation of formulas that transforms each of
the logical entropy compound formulas into the corresponding Shannon entropy compound formula,
and the transform preserves the Venn diagram relationships that automatically hold for measures.
This "dit-bit transform" is heuristically motivated by showing how average counts of distinctions
("dits") can be converted in average counts of binary partitions ("bits").
Moreover, Campbell remarked that it would be "particularly interesting" and "quite signi…cant"
if there was an entropy measure of sets so that joint entropy corresponded to the measure of the
union of sets, conditional entropy to the di¤erence of sets, and mutual information to the intersection
of sets [9, p. 113]. Logical entropy precisely satis…es those requirements.
2 Logical information as the measure of distinctions
There is now a widespread view that information is fundamentally about di¤erences, distinguisha-
bility, and distinctions. As Charles H. Bennett, one of the founders of quantum information theory,
put it:
So information really is a very useful abstraction. It is the notion of distinguishability
abstracted away from what we are distinguishing, or from the carrier of information. [5,
p. 155]
This view even has an interesting history. In James Gleick’s book, The Information: A History,
A Theory, A Flood, he noted the focus on di¤erences in the seventeenth century polymath, John
Wilkins, who was a founder of the Royal Society. In 1641, the year before Isaac Newton was born,
Wilkins published one of the earliest books on cryptography, Mercury or the Secret and Swift Mes-
senger, which not only pointed out the fundamental role of di¤erences but noted that any (…nite)
set of di¤erent things could be encoded by words in a binary code.
For in the general we must note, That whatever is capable of a competent Di¤erence,
perceptible to any Sense, may be a su¢ cient Means whereby to express the Cogitations.
It is more convenient, indeed, that these Di¤erences should be of as great Variety as the
Letters of the Alphabet; but it is su¢ cient if they be but twofold, because Two alone
may, with somewhat more Labour and Time, be well enough contrived to express all the
rest. [47, Chap. XVII, p. 69]
Wilkins explains that a …ve letter binary code would be su¢ cient to code the letters of the alphabet
since 2
5
= 32.
Thus any two Letters or Numbers, suppose A:B. being transposed through …ve Places,
will yield Thirty Two Di¤erences, and so consequently will superabundantly serve for
the Four and twenty Letters... .[47, Chap. XVII, p. 69]
3

As Gleick noted:
Any di¤erence meant a binary choice. Any binary choice began the expressing of cogi-
tations. Here, in this arcane and anonymous treatise of 1641, the essential idea of infor-
mation theory poked to the surface of human thought, saw its shadow, and disappeared
again for [three] hundred years. [20, p. 161]
Thus counting distinctions [12] would seem the right way to measure information,
1
and that is the
measure which emerges naturally out of partition logic–just as …nite logical probability emerges
naturally as the measure of counting elements in Boole’s subset logic.
Although usually named after the special case of ‘propositional’logic, the general case is Boole’s
logic of subsets of a universe U (the special case of U = 1 allows the propositional interpretation
since the only subsets are 1 and ; standing for truth and falsity). Category theory shows there is a
duality between sub-sets and quotient-sets (= partitions = equivalence relations), and that allowed
the recent development of the dual logic of partitions ([13], [15]). As indicated in the title of his
book, An Investigation of the Laws of Thought on which are founded the Mathematical Theories of
Logic and Probabilities [7], Boole also developed the normalized counting measure on subsets of a
…nite universe U which was …nite logical probability theory. When the same mathematical notion
of the normalized counting measure is applied to the partitions on a …nite universe set U (when the
partition is represented as the complement of the corresponding equivalence relation on U  U) then
the result is the formula for logical entropy.
In addition to the philosophy of information literature [4], there is a whole sub-industry in
mathematics concerned with di¤erent notions of ‘entropy’or ‘information’([2]; see [45] for a recent
‘extensive’ analysis) that is long on formulas and ‘intuitive axioms’ but short on interpretations.
Out of that plethora of de…nitions, logical entropy is the measure (in the technical sense of measure
theory) of information that arises out of partition logic just as logical probability theory arises out
of subset logic.
The logical notion of information-as-distinctions supports the view that the notion of information
is independent of the notion of probability and should be based on …nite combinatorics. As Andrei
Kolmogorov put it:
Information theory must precede probability theory, and not be based on it. By the very
essence of this discipline, the foundations of information theory have a …nite combinato-
rial character. [27, p. 39]
Logical information theory precisely ful…lls Kolmogorov’s criterion.
2
It starts simply with a set
of distinctions de…ned by a partition on a …nite set U , where a distinction is an ordered pair of
elements of U in distinct blocks of the partition. Thus the “…nite combinatorial” object is the
set of distinctions ("ditset") or information set ("infoset") associated with the partition, i.e., the
complement in U  U of the equivalence relation associated with the partition. To get a quantitative
measure of information, any probability distribution on U de…nes a product probability measure on
U  U, and the logical entropy is simply that probability measure of the information set.
3 Duality of subsets and partitions
Logical entropy is to the logic of partitions as logical probability is to the Boolean logic of sub-
sets. Hence we will start with a brief review of the relationship between these two dual forms of
1
This paper is about what Adriaa ns and van Benthem call "Information B: Probabilistic, information-theoretic,
measure d quantitatively", not about "Information A: knowledge, logic, wha t is conveyed in i nformative answers" where
the co nnection t o philosophy and logic is built-in from the beginning. Likewise, the p aper is not about Kolmogorov-
style "Information C: Algorithmic, code compression, mea sured quantitatively. " [4, p. 11]
2
Ko lmogorov had something else in mind such as a co mbinatorial development of Hart ley’s log (n) on a s et of n
equiprobable elements.[28]
4

Frequently asked questions

Q1. What are the contributions in this paper?

In this manner, logical information theory provides the set-theoretic and measure-theoretic foundations for information theory.