Posted Content•
The Generalized Universal Law of Generalization
TL;DR: In this paper, the authors show that the universal law of generalization holds with probability going to one-provided the confusion probabilities are computable, and they also give a mathematically more appealing form.
Abstract: It has been argued by Shepard that there is a robust psychological law that relates the distance between a pair of items in psychological space and the probability that they will be confused with each other. Specifically, the probability of confusion is a negative exponential function of the distance between the pair of items. In experimental contexts, distance is typically defined in terms of a multidimensional Euclidean space-but this assumption seems unlikely to hold for complex stimuli. We show that, nonetheless, the Universal Law of Generalization can be derived in the more complex setting of arbitrary stimuli, using a much more universal measure of distance. This universal distance is defined as the length of the shortest program that transforms the representations of the two items of interest into one another: the algorithmic information distance. It is universal in the sense that it minorizes every computable distance: it is the smallest computable distance. We show that the universal law of generalization holds with probability going to one-provided the confusion probabilities are computable. We also give a mathematically more appealing form
Citations
More filters
Dissertation•
01 Nov 2007TL;DR: In this article, the authors give thanks to many people who have provided support, help and encouragement over the last couple of years, without them this thesis would not have materialized, and they need to be recognized.
Abstract: ii To my parents. I am indebted to many people who have provided support, help and encouragement over the last couple of years—without them this thesis would not have materialized. Above all I need to thank Felix Wichmann for his supervision and thoughtful advice on all aspects of this work, for teaching me how to run proper psychophysical experiments, and for the right encouragement at the right time. Very special thanks go to Bernhard Schölkopf for introducing me to kernel methods. Without his vision and without his support I would not have started working on this topic. It was a great pleasure and a great opportunity to work in his department that offers a unique and truly interdisciplinary environment for studying learning and perception in man and machine. Many thanks go to all members of AGBS for making the department a place where ideas and insights are happily shared and for making it a fun place to work at. Theresa Cooke, Dilan Görür, Malte Kuss, and I worked together on various projects. Steinke were kind enough to read large parts of this thesis at various stages and provided many helpful comments.
1 citations
TL;DR: It is proposed that social categorization is driven by an ecological dynamic that operates in two planes: feature space and label space, and a theoretical model that links positions in the two spaces is developed that predicts that an actor’s proximity in feature space to a labeled cluster increases its propensity to adopt the label.
Abstract: This paper proposes that social categorization is driven by an ecological dynamic that operates in two planes: feature space and label space. It develops a theoretical model that links positions in the two spaces. The first part of the theory predicts that an actor’s proximity in feature space to a labeled cluster increases its propensity to adopt the label. The second part predicts that the structure of label space affects this relationship: featurespace positions are more weakly related to labeling for lenient labels. An empirical analysis of software producers, based on their positions in a technical feature space (derived from portfolios of patents) and a space of market labels, supports these predictions. The results imply that one producer’s changed labeling can change the sets of feature values associated with a label, which then affects other producers in the domain. This coupled ecological dynamic is found even in a loosely governed system of self-categorization. Findings suggest that constraining categories become more constraining, while lenient categories become more lenient. When people judge whether an olive is a fruit or whether Pluto is a planet, their assessments do not depend on what the fruits and celestial bodies have to say about the matter. The situation differs sharply for categorization of humans and corporate actors. These kinds of agents can, and often do, dispute categorical assignments. Re-classification of one object can affect not only how it gets perceived but also the meaning of the category. We are struck by the possibility that this latter effect produces an ecological dynamic for social categorization. Classification of one entity, say a cultural product, affects the classification of others, potentially generating cascades of changes. This ecological dynamic operates in two planes: one involves feature values and the other involves profiles of label assignments. The two planes are connected through categorization when agents label objects. Sometimes labels are linked to feature values that are common to a set of objects. This is the first step to the label becoming infused with social meaning.1 The association between labels and feature values is straightforward in a static world. But social actors, unlike olives and other natural kinds, can change their characteristics. Moves in feature space by members of a category can make atypical Date: July 24, 2012; revised December 7, 2012. We appreciate the extremely helpful suggestions of Greta Hsu, Balazs Kovacs, Gael Le Mens, and Laszlo Polos. This study was supported by the Chicago-Booth Graduate School of Business, the Stanford Graduate School of Business, and the John Osterweis and Barbara Ravizzi Faculty Fellowship at the Stanford GSB. 1In our construction a feature is a dimension such as the form of authority in an organization. Positions in feature space are constructed from values of features, e.g., tradition, rational-legal, or charismatic forms of authority.
1 citations
01 Jan 2007
TL;DR: Own designed gauges - the average overlap and robustness of description coefficients - are presented to analyze the structure of emerging communities encompassing similar data.
Abstract: In this paper, we tackle the issues of analyzing the structural
evolution of the metric social network. The metric social
network operates in a P2P environment where peers maintain
their own data and the relationships among them are formed on
the basis of the processed similarity queries. The evolution is
analyzed by traditional social networking tools - the
characteristic path length and the clustering coefficient.
Nonetheless, due to the special structure of the metric social
network, own designed gauges - the average overlap and
robustness of description coefficients - are presented to
analyze the structure of emerging communities encompassing
similar data.
Dissertation•
01 Jan 2010
TL;DR: In this paper, structural alignment (SA) and representational distortion (RD) are used as a general framework for understanding the similarity between object representations, and a detailed set of transformational predictions (coding scheme) within a rich stimulus domain is investigated.
Abstract: Similarity, being a psychological notion, involves the comparison of finite object representations. The specific nature and complexity of these representations is a matter of fierce theoretical debate traditionally, similarity research was dominated by the spatial and featural account. In the spatial account, similarity is determined by the distance between objects in a psychological space. Alternatively, the featural account proposes that similarity is determined by matching objects' features. Despite the empirical success of these accounts, the object representations they posit are regarded too simple and specific to deal with more complex objects. Therefore, two structural accounts have been developed: structural alignment (SA) and Representational Distortion (RD). This aim of this thesis was to further establish one particular structural account RD as a general framework for understanding the similarity between object representations. Specifically, RD measures similarity by the complexity of the transformation that "distorts" one representation into the other. This RD approach is investigated in detail by testing a detailed set of transformational predictions (coding scheme) within a rich stimulus domain. These predictions are tested through experiments and modelling that utilise both a) explicit measures (ratings, forced-choice), and, for the first time, b) implicit measures (reaction time, same-different errors & spontaneous categorisation). Moreover, RD is compared empirically with both traditional and alignment models of similarity. Overall, the results suggest that similarity can be best understood by transformational relationships in a number of contexts. The performance of RD in both explicit and implicit measures is made more compelling by the fact that rival accounts fundamentally struggle to describe the sorts of relationships that are easily captured by RD. Finally, it is emphasised that RD is actually compatible with supposedly rival approaches and can incorporate theoretically these accounts, both traditional and structural, under one general framework.
References
More filters
Book•
01 Jan 1991TL;DR: The author examines the role of entropy, inequality, and randomness in the design of codes and the construction of codes in the rapidly changing environment.
Abstract: Preface to the Second Edition. Preface to the First Edition. Acknowledgments for the Second Edition. Acknowledgments for the First Edition. 1. Introduction and Preview. 1.1 Preview of the Book. 2. Entropy, Relative Entropy, and Mutual Information. 2.1 Entropy. 2.2 Joint Entropy and Conditional Entropy. 2.3 Relative Entropy and Mutual Information. 2.4 Relationship Between Entropy and Mutual Information. 2.5 Chain Rules for Entropy, Relative Entropy, and Mutual Information. 2.6 Jensen's Inequality and Its Consequences. 2.7 Log Sum Inequality and Its Applications. 2.8 Data-Processing Inequality. 2.9 Sufficient Statistics. 2.10 Fano's Inequality. Summary. Problems. Historical Notes. 3. Asymptotic Equipartition Property. 3.1 Asymptotic Equipartition Property Theorem. 3.2 Consequences of the AEP: Data Compression. 3.3 High-Probability Sets and the Typical Set. Summary. Problems. Historical Notes. 4. Entropy Rates of a Stochastic Process. 4.1 Markov Chains. 4.2 Entropy Rate. 4.3 Example: Entropy Rate of a Random Walk on a Weighted Graph. 4.4 Second Law of Thermodynamics. 4.5 Functions of Markov Chains. Summary. Problems. Historical Notes. 5. Data Compression. 5.1 Examples of Codes. 5.2 Kraft Inequality. 5.3 Optimal Codes. 5.4 Bounds on the Optimal Code Length. 5.5 Kraft Inequality for Uniquely Decodable Codes. 5.6 Huffman Codes. 5.7 Some Comments on Huffman Codes. 5.8 Optimality of Huffman Codes. 5.9 Shannon-Fano-Elias Coding. 5.10 Competitive Optimality of the Shannon Code. 5.11 Generation of Discrete Distributions from Fair Coins. Summary. Problems. Historical Notes. 6. Gambling and Data Compression. 6.1 The Horse Race. 6.2 Gambling and Side Information. 6.3 Dependent Horse Races and Entropy Rate. 6.4 The Entropy of English. 6.5 Data Compression and Gambling. 6.6 Gambling Estimate of the Entropy of English. Summary. Problems. Historical Notes. 7. Channel Capacity. 7.1 Examples of Channel Capacity. 7.2 Symmetric Channels. 7.3 Properties of Channel Capacity. 7.4 Preview of the Channel Coding Theorem. 7.5 Definitions. 7.6 Jointly Typical Sequences. 7.7 Channel Coding Theorem. 7.8 Zero-Error Codes. 7.9 Fano's Inequality and the Converse to the Coding Theorem. 7.10 Equality in the Converse to the Channel Coding Theorem. 7.11 Hamming Codes. 7.12 Feedback Capacity. 7.13 Source-Channel Separation Theorem. Summary. Problems. Historical Notes. 8. Differential Entropy. 8.1 Definitions. 8.2 AEP for Continuous Random Variables. 8.3 Relation of Differential Entropy to Discrete Entropy. 8.4 Joint and Conditional Differential Entropy. 8.5 Relative Entropy and Mutual Information. 8.6 Properties of Differential Entropy, Relative Entropy, and Mutual Information. Summary. Problems. Historical Notes. 9. Gaussian Channel. 9.1 Gaussian Channel: Definitions. 9.2 Converse to the Coding Theorem for Gaussian Channels. 9.3 Bandlimited Channels. 9.4 Parallel Gaussian Channels. 9.5 Channels with Colored Gaussian Noise. 9.6 Gaussian Channels with Feedback. Summary. Problems. Historical Notes. 10. Rate Distortion Theory. 10.1 Quantization. 10.2 Definitions. 10.3 Calculation of the Rate Distortion Function. 10.4 Converse to the Rate Distortion Theorem. 10.5 Achievability of the Rate Distortion Function. 10.6 Strongly Typical Sequences and Rate Distortion. 10.7 Characterization of the Rate Distortion Function. 10.8 Computation of Channel Capacity and the Rate Distortion Function. Summary. Problems. Historical Notes. 11. Information Theory and Statistics. 11.1 Method of Types. 11.2 Law of Large Numbers. 11.3 Universal Source Coding. 11.4 Large Deviation Theory. 11.5 Examples of Sanov's Theorem. 11.6 Conditional Limit Theorem. 11.7 Hypothesis Testing. 11.8 Chernoff-Stein Lemma. 11.9 Chernoff Information. 11.10 Fisher Information and the Cram-er-Rao Inequality. Summary. Problems. Historical Notes. 12. Maximum Entropy. 12.1 Maximum Entropy Distributions. 12.2 Examples. 12.3 Anomalous Maximum Entropy Problem. 12.4 Spectrum Estimation. 12.5 Entropy Rates of a Gaussian Process. 12.6 Burg's Maximum Entropy Theorem. Summary. Problems. Historical Notes. 13. Universal Source Coding. 13.1 Universal Codes and Channel Capacity. 13.2 Universal Coding for Binary Sequences. 13.3 Arithmetic Coding. 13.4 Lempel-Ziv Coding. 13.5 Optimality of Lempel-Ziv Algorithms. Compression. Summary. Problems. Historical Notes. 14. Kolmogorov Complexity. 14.1 Models of Computation. 14.2 Kolmogorov Complexity: Definitions and Examples. 14.3 Kolmogorov Complexity and Entropy. 14.4 Kolmogorov Complexity of Integers. 14.5 Algorithmically Random and Incompressible Sequences. 14.6 Universal Probability. 14.7 Kolmogorov complexity. 14.9 Universal Gambling. 14.10 Occam's Razor. 14.11 Kolmogorov Complexity and Universal Probability. 14.12 Kolmogorov Sufficient Statistic. 14.13 Minimum Description Length Principle. Summary. Problems. Historical Notes. 15. Network Information Theory. 15.1 Gaussian Multiple-User Channels. 15.2 Jointly Typical Sequences. 15.3 Multiple-Access Channel. 15.4 Encoding of Correlated Sources. 15.5 Duality Between Slepian-Wolf Encoding and Multiple-Access Channels. 15.6 Broadcast Channel. 15.7 Relay Channel. 15.8 Source Coding with Side Information. 15.9 Rate Distortion with Side Information. 15.10 General Multiterminal Networks. Summary. Problems. Historical Notes. 16. Information Theory and Portfolio Theory. 16.1 The Stock Market: Some Definitions. 16.2 Kuhn-Tucker Characterization of the Log-Optimal Portfolio. 16.3 Asymptotic Optimality of the Log-Optimal Portfolio. 16.4 Side Information and the Growth Rate. 16.5 Investment in Stationary Markets. 16.6 Competitive Optimality of the Log-Optimal Portfolio. 16.7 Universal Portfolios. 16.8 Shannon-McMillan-Breiman Theorem (General AEP). Summary. Problems. Historical Notes. 17. Inequalities in Information Theory. 17.1 Basic Inequalities of Information Theory. 17.2 Differential Entropy. 17.3 Bounds on Entropy and Relative Entropy. 17.4 Inequalities for Types. 17.5 Combinatorial Bounds on Entropy. 17.6 Entropy Rates of Subsets. 17.7 Entropy and Fisher Information. 17.8 Entropy Power Inequality and Brunn-Minkowski Inequality. 17.9 Inequalities for Determinants. 17.10 Inequalities for Ratios of Determinants. Summary. Problems. Historical Notes. Bibliography. List of Symbols. Index.
45,034 citations
01 Jan 1979
42,654 citations
01 Jan 1979
12,336 citations
Book•
01 Jan 1948
TL;DR: The Mathematical Theory of Communication (MTOC) as discussed by the authors was originally published as a paper on communication theory more than fifty years ago and has since gone through four hardcover and sixteen paperback printings.
Abstract: Scientific knowledge grows at a phenomenal pace--but few books have had as lasting an impact or played as important a role in our modern world as The Mathematical Theory of Communication, published originally as a paper on communication theory more than fifty years ago. Republished in book form shortly thereafter, it has since gone through four hardcover and sixteen paperback printings. It is a revolutionary work, astounding in its foresight and contemporaneity. The University of Illinois Press is pleased and honored to issue this commemorative reprinting of a classic.
10,215 citations
TL;DR: This chapter discusses the application of the diagonal process of the universal computing machine, which automates the calculation of circle and circle-free numbers.
Abstract: 1. Computing machines. 2. Definitions. Automatic machines. Computing machines. Circle and circle-free numbers. Computable sequences and numbers. 3. Examples of computing machines. 4. Abbreviated tables Further examples. 5. Enumeration of computable sequences. 6. The universal computing machine. 7. Detailed description of the universal machine. 8. Application of the diagonal process. Pagina 1 di 38 On computable numbers, with an application to the Entscheidungsproblem A. M. ...
7,642 citations