New Articles...



The Confused and Confusing Nature of Information

It is a very well known and documented fact and problem not only that the term 'information' is not only applied to many different ideas, dynamics, quantities, measures, and concepts across science and philosophy (not to mention every day usage), but that even in the context of central scientific theories that constitute the locus classicus of the information theory, it is not clear exactly what is being referred to using the term.

In a famous anecdote conveyed to his students by Claude Elwood Shannon, Shannon relates that in a conversation with Von Neumann that he is perplexed about what to call what he is measuring. Von Neumann quipped that he should call it entropy, because of the ambiguity surrounding that term in physics:

However, there is a very basic error that continues to repeat itself in the literature, which error reveals that the confusion is far from over, or even alleviated. I say error, because from the perspective of the relevant philosophical disciplines of both philosophical and formal logic and metaphysics, it clearly involves a category error, at best. The error involves the confusion of measures of information with information itself. Put otherwise, the term 'information' is used to refer to whatever is being measured, as well as to the measure itself, and also often to the quantity identified by the measure. Ontologically, a measure is not that which is measured, nor even necessarily is it the magnitude thus delivered. In the case of information it is like saying that information is a measure of information. This involves a basic circularity, as well as a category error.

While it is true that in speaking about information as some kind of content or commodity that can increase knowledge under the right conditions (one of the many available definitions of information itself - Drestkse, 1981; Adami, 2015), a measure of information may well provide further or other information: it does not follow that the nature of information is that it exists as something in the ontology, and as a magnitude or measure associated with that thing. Even if ontological eliminitavists about information are right and there is no such independently existing type of thing in the world as information, and the term 'information' is just one that we apply to various measures and characterisations of various other things such as statistical uncertainty, or entropy, or epistemic content: information measures still cannot be identical to whatever it is that they are measuring or are a measure of. Many analyses of the nature of information and of the concepts of information in Shannon's theory (and other loci classici of algorithmic and statistical information theory and computational theories of information) attempt to address the question of the nature of information (see the entire reference list, at least). What frequently happens is that the author of what is usually a scientifically informed (with reference to the best applied mathematical information theory) conceptual analysis refers to both that which is being measured and the measure of it as information.

One of the world's foremost experts on information theory in the sciences - the multidisciplinary genius Christoph Adami - has struggled with the ambiguities surrounding Claude Shannon's classical mathematical theory in this way. In his excellent Information theory in molecular biology (Adami, 2004) he astutely observes that:

"Entropy and information, the two central concepts of Shannon’s theory of information and communication, are often confused with each other but play transparent roles when applied to statistical ensembles (i.e., identically prepared sets) of symbolic sequences. Such an approach can distinguish between entropy and information in genes...I outline the concepts of entropy and information (as defined by Shannon) in the context of molecular biology. We shall see that not only are these terms well-defined and useful, they also coincide precisely with what we intuitively mean when we speak about information stored in genes, for example. I then present examples of applications of the theory to measure the information content of biomolecules...Entropy and information are often used in conflicting manners in the literature. A precise understanding,both mathematical and intuitive, of the notion of information (and its relationship to entropy) iscrucial for applications in molecular biology...How do we ever learn anything about a system? There are two choices. Either we obtain the probability distribution using prior knowledge (for example, by taking the system apart and predicting its states theoretically) or by making measurements on it, which for example might reveal that not all states, infact, are taken on with the same probability. In both cases, the difference between the maximal entropy and the remaining entropy after we have either done our measurements or examined the system, is the amount of information we have about the system. Before..." (4-5)

I have excised a lot from the above, and Adami's analysis and adaptation of the classical statistical model to his purpose is quite masterful. Nonetheless, then the trouble starts in earnest:

"Before I write this into a formula, let me remark that, by its very definition, information is a relative quantity. It measures the difference of uncertainty, in the previous case the entropy before and after the measurement, and thus can never be absolute, in the same sense as potential energy in physics is not absolute. In fact, it is not a bad analogy to refer to entropy as “potential information”, because potentially all of a system’s entropy can be transformed into information (for example by measurement)...In the above case, information was the difference between the maximal and the actual entropy of a system. This is not the most general definition as I have alluded to. More generally, information measures the amount of correlation between two systems, and reduces to a difference in entropies in special cases. To define information properly, let me introduce another random variable or molecule (call it “Y ”), which can be in states y1, . . ., yM with probabilities p1, . . .,pM. We can then, along with the entropy H(Y), introduce the joint entropy H(XY), which measures my uncertainty about the joint system XY (which can be in N ·M states). If X and Y are independent random variables (like, e.g., two dice that are thrown independently) the joint entropy will be just the sum of the entropy of each of the random variables. Not so if X and Y are somehow connected. Imagine, for example, two coins that are glued together at one face. Then, heads for one of the coins will always imply tails for the other, and vice versa. By gluing them together, the two coins can only take on two states, not four, and the joint entropy is equal to the entropy of one of the coins." (5-6)

So, now we must ask ourselves - terminological habits of working applied mathematicians and scientists aside - is information a quantity, a measure of a quantity, a measure of differences between quantities, an abstract representation of a measure, an abstract representation of a quantity - or something else in the measured system? If it is a quantity or a measure - then how is it content? Content is usually not thought of in terms of just a magnitude. It usually is taken to exist as something that is one or more of semantic, functional, structural, and has properties. Is it a subjective measure of subjective uncertainty - or an objective measure of frequentist uncertainty (not the same thing at all)?

I do not mean to be unfair to Adami here. He is taking on what I suggest is one of the most difficult conceptual and definitional tasks in the history of science, and he is by no means the only one to struggle thus. The nature of information, quantum information, and biological information have troubled some of the best minds across multiple disciplines for the last 75 years (Rolf Landauer, Shannon himself, and Norbert Weiner all struggled with the same issues.)

One could be forgiven for suspecting that my observation about the confusion of information measures with that which they are taken to measure involves uninformed (in terms of scientific and mathematical praxis) and unnecessary pedantry. After all - applied mathematicians, scientists, and engineers frequently semantically overload their terms, using the same term to refer to the physical quantity or observable and the abstract mathematical representation of it (observables in quantum mechanics, and random variables, events, populations, and samples in statistics ). In Shannon's theory, the term 'source' does just this kind of double duty - naming a physical stochastic process and the model-representation thereof. However, scholars that approach the question of the nature of information and what the term 'information' is referring to in information theory, physics, and molecular bioscience, do so with the understanding that the pedantry and exactitude is exactly what is required to address the problem of ambiguity. Essentially - they intend to, and are, doing philosophy in the form of either scientific metaphysics or conceptual analyses, and they know it.

Thus, in the philosophy of information and in information theory, confusing the measure of anything with that which is being measured results in a category error. In the case of information theory and the philosophy of information this is not tolerable, since correctly determining the nature of information is a primary concern, and ambiguity and confusion surrounding the nature of information has led some scholars beyond pluralism about the nature of information and into eliminativism or anti-realism about it in some scientific contexts (Griffiths, 2001).

Bibliography and References


  2. Adami, Christoph. (2004). Information theory in molecular biology. Physics of Life Reviews, 1(1), 3–22.

  3. Adriaans, P. (2009). Between order and chaos: The quest for meaningful information. Theory of Computing Systems, 45(4), 650–674.

  4. Adriaans, P. (2010). A Critical Analysis of Floridi’s Theory of Semantic Information, 1–16.

  5. Artmann, S. (2008). Biological Information (pp. 22–39).

  6. Barbieri, M. (2012). What is Information? Biosemiotics, 5(2), 147–152.

  7. Caticha, A. (2014). Towards an Informational Pragmatic Realism. Minds and Machines, 24(1), 37–70.

  8. Chaitin, G. J. (1987). Algorithmic information theory (Vol. 1). Cambridge [Cambridgeshire];New York; Cambridge University Press.

  9. Devlin, K. J. (1991). Logic and information. New York; Cambridge [England]: Cambridge University Press.

  10. DiVincenzo, D. P., & Loss, D. (1998). Quantum information is physical. Superlattices and Microstructures, 23(3), 419–432.

  11. Floridi, L. (2005). Is Semantic Information Meaningful Data? Philosophy and Phenomenological Research, 70(2), 351–370.

  12. Floridi, L. (2009a). Against Digital Ontology. Synthese, 168(1), 151–178.

  13. Floridi, L. (2009b). Philosophical conceptions of information. In Formal theories of information: from shannon to semantic information theory and general concepts of information (Vol. 5363, pp. 13–53).

  14. Floridi, L. (2011). The philosophy of information. Oxford; New York: Oxford University Press.

  15. Griffiths, P. E. (2001). Genetic Information: A Metaphor in Search of a Theory. Philosophy of Science, 68(3), 394–412.

  16. Grünwald, P. D., & Vitányi, P. M. B. (2008). Algorithmic Information Theory (pp. 281–317).

  17. Hamame, C. M., Cosmelli, D., & Aboitiz, F. (2007). What is so informative about information? Behavioral and Brain Sciences, 30(4), 371–372.

  18. Harms, W. F. (2006). What Is Information? Three Concepts. Biological Theory, 1(3), 230–242.

  19. Ibekwe-SanJuan, F., & Dousa, T. M. (2013). Theories of Information, Communication and Knowledge: A Multidisciplinary Approach (1;2014;, Vol. 34). Dordrecht: Springer Netherlands.

  20. Ladyman, J., & Ross, D. (2013). The world in the data. In Scientific Metaphysics. Oxford University Press, Oxford. Oxford University Press.

  21. Landauer, R. (1996). The physical nature of information. Physics Letters A, 217(4), 188–193.

  22. Landauer, R. (1999). Information is a physical entity. Physica A: Statistical Mechanics and Its Applications, 263(1), 63–67.

  23. Lombardi, O. (2004). What is Information? Foundations of Science, 9(2), 105–134.

  24. Long, B. R. (2014). Information is intrinsically semantic but alethically neutral. Synthese, 191(14), 3447–3467.

  25. McIrvine, E. C., & Tribus, M. (1971). Energy and Information. Scientific American, 225(3), 179–88.

  26. Rocchi, P. (2011). Ubiquity symposium: What is information?: beyond the jungle of information theories. Ubiquity, 2011(March), 1–9.

  27. Rovelli, C. (2016). Meaning = Information + Evolution.

  28. Rowley, J. (1998). What is information? Information Services & Use, 18(4), 243.

  29. Shannon, C. (1956). The bandwagon (Edtl.). IRE Transactions on Information Theory, 2(1), 3–3.

  30. Timpson, C. G. (2013). What is Information? Oxford: Oxford University Press.

  31. Tribus, M. (1963). Information Theory and Thermodynamics. In Heat Transfer, Thermodynamics, and Education: Boelter Anniversary Volume (pp. 348–68). McGraw Hill.

Links of Interest

Christoph Adami at The Conversation


  • twitter
  • facebook

©2016-2019 by informationist.