The overall objectives of this review and synthesis are to study the basics of information-theoretic methods in econometrics, to examine the connecting theme among these methods, and to provide a more detailed summary and synthesis of the sub-class of methods that treat the observed sample moments as stochastic. Within the above objectives, this review focuses on studying the inter-connection between information theory, estimation, and inference. To achieve these objectives, it provides a detailed survey of information-theoretic concepts and quantities used within econometrics. It also illustrates the use of these concepts and quantities within the subfield of information and entropy econometrics while paying special attention to the interpretation of these quantities. The relationships between information-theoretic estimators and traditional estimators are discussed throughout the survey. This synthesis shows that in many cases information-theoretic concepts can be incorporated within the traditional likelihood approach and provide additional insights into the data processing and the resulting inference.

/pdf/information-and-entropy-econometrics-a-review-and-synthesis-286rrggbao.pdf

Information and Entropy Econometrics - A Review and Synthesis

(2000). Principal Information Theoretic Approaches. Journal of the American Statistical Association: Vol. 95, No. 452, pp. 1349-1353.

Principal Information Theoretic Approaches

http://fs2.american.edu/agolan/www/infoentropy.pdf

Information and Entropy Econometrics—Editor's View

Information indices: unification and applications

Information-theoretic methodologies are increasingly being used in various disciplines. Fre- quently an information measure is adapted for a problem, yet the perspective of information as the unifying notion is overlooked. We set forth this perspective through presenting information- theoretic methodologies for a set of problems in probability and statistics. Our focal measures are Shannon entropy and Kullback-Leibler information. The background topics for these mea- sures include notions of uncertainty and information, their axiomaticfoundation, interpretations, properties, and generalizations. Topics with broad methodological applications include discrep- ancy between distributions, derivation of probability models, dependence between variables, and Bayesian analysis. More specific methodological topics include model selection, limiting distributions, optimal prior distribution and design of experiment, modeling duration variables, order statistics, data disclosure, and relative importance of predictors. Illustrations range from very basic to highly technical ones that draw attention to subtle points.

Information Measures in Perspective

The Cramer-Rao inequality provides, under certain regularity conditions, a lower bound for the variance of an estimator [7], [15]. Various generalizations, extensions and improvements in the bound have been made, by Barankin [1], [2], Bhattacharyya [3], Chapman and Robbins [5], Fraser and Guttman [11], Kiefer [12], and Wolfowitz [16], among others. Further considerations of certain inequality properties of a measure of information, discussed by Kullback and Leibler [14], yields a greater lower bound for the information measure (formula (4.11)), and leads to a result which may be considered a generalization of the Cramer-Rao inequality, the latter following as a special case. The results are used to define discrimination efficiency and estimation efficiency at a point in parameter space.

/pdf/certain-inequalities-in-information-theory-and-the-cramer-4x2o7afu4h.pdf

S. Kullback

Papers

Certain Inequalities in Information Theory and the Cramer-Rao Inequality