INFORMATION THEORY


Sections:

  • Oyama - The Ontogeny of Information
  • Ashby - Introduction to Cybernetics, Design for a Brain
  • Artifical Intelligence

      The study of living systems is the study of change, of process, of development. Central to understanding the process of biological change is the notion of "information." Despite the well-worked out concepts of information in communications engineering, the notion is still going through a difficult period of adaptation as a useful concept for understanding living systems.

      Information theory was elaborated in the area of communications engineering as part of a solution to achieve maximum efficiency of telephone communications. The resulting mathematical theory of communication was invaluable when rigourously applied to communication problems in engineering, but it has been loosely extended to all communication by living organisms (Shannon, C. & Weaver, W.(1949). A Mathematical Model of Communication. Urbana, IL; University of Illinois Press). An argument may be made that living organisms differ in significant ways from the relatively simple systems that communication theorists were originally working with. Hence, the utility of the information-theory account for research in the life sciences must be re-examined, along with the ways in which it has been extended to living organisms.

      Information theory separated the concept of information from thermodynamic considerations as it had been used in physics. Used in statistical treatments of messages moving through channels from sources to receivers

      Oyama, S. (1985). The Ontogeny of Information: Developmental Systems and Evolution. Cambridge; Cambridge University Press.

      1. Information as communication:

      "What is fundamental to the communications concept of information is its purely statistical nature; content and meaning are not considerations, but predictability is. The amount of information transmitted, that is, depends on the number of alternatives; an event that reduces no uncertainty carries no information. In the same way that a variable becomes a determinant when it accounts for a difference in outcome, a message conveys information when it distinguishes one possibility from another (p. 65, italics added)."
      When the system is well-defined, then quantification may be possible. Those factors which do not vary can "neither account for variability nor convey information" (compare to experimental design). The possibility of variation, limits on variation, is constrained by the factors held constant, like controlled variables in an experiment. Constraints reduce degrees of freedom, thereby limiting the variability of variables.

      Information Theory and Biological Systems

      In biology, this statistical or probability-based interpretation of information is rarely present, and even less so in the human sciences. Communication theory is generally invoked in a very loose manner. For example, in a living organism it is not clearly evident what should be defined as the source, channel and receiver. The source is that which selects symbols from a set and encodes a message. We could argue that any level of analysis constitutes the source - the "zygote...whose DNA is assembled at fertilization, or that the process of reproduction, or the species or the gene pool is the source. The channel might be ontogeny itself."

      Biological systems are open systems, unlike the physical systems dealt with by communication theorists. This was an important conceptualization in trying to explain the presence of living systems (order-producing) in a universe that was moving toward entropy. What "open system" implies here is that the "system" consists of multiple levels - organism and its environment. The distinction between organism and environment is blurred...as is the distinction between "levels."

      In the relatively closed systems of the information theorist, information may be degraded but not increased. In ontogeny, the system is an open one in which energy is dissipated but complexity increased.

      "The engineer's message can only be harmed or lost by transmission, while the ontogenetic "message" is created in transmission (p. 65)."
      The receiver's characteristics are an important determinant of the amount of information transmitted. "The significance of any information influencing developmental processes, in other words, whether from genetic transcription or from other events, is defined by the developing system itself."

      Information is "a difference that makes a difference."

      Information theory is very commonly invoked today in explaining many things - from the physical universe to computers to the brain. It has become a metaphor. In the first part of this section Oyama presents some examples of its use by investigators.

      Information theory has become a part of our vocabulary, but it has fit in well with previous categories of explanation.

      "One of the problems I will address then, is the idea of developmental information in the chromosomes. What does it explain? What is it information about? What important questions does it or does it not clarify? When is it invoked metaphorically and what could it mean to treat it literally? When, if ever, does it really help us understand how the living world is constituted and what the prospects for that world are? (p. 5)"
      We often speak today as if "information" were a third force in the world, with equal status to energy and matter. "Both the initiation and course of biological change are a function of developmental systems, and there is no evidence that our notions of matter and energy exchanges, themselves admittedly evolving, are inadequate to describe them. Adding information to matter and energy is something like speaking of nations exchanging dollars, yen and profits. The third term belongs on a different level. Not another currency, it describes a certain disposition and use of currencies. Just as time or information can, under certain circumstances, "be" money, matter and energy can sometimes "be" information (p. 34-35)."
      2. Information as causal control: variations and constraints
      (as cause and contingency of) "At present, all that needs to be pointed out is that these exchanges often suggest, whatever the other philosophical biases of the speaker, a "preformationist" attitude toward information. It exists before its utilization or expression. Some views allow assembly of information from a variety of sources, but this in turn implies that it exists in several loci before being collected; such views thus perpetuate the mistake while seeming to correct it. In addition, information is conceived to be a special kind of cause among all the factors that may be necessary for a phenomenon, the cause that imparts order and form to matter. The alternative to such a preformationist attitude toward form is not a classical epigenetic one. Not only did this traditionally require that order arise from chaos, an unsatisfactory solution at best, but it often posited a vitalistic force as well, to effect the recurrent miracle. This brings us full circle to the preexisting form, this time ready to inform the formless, rather than simply waiting to unfold. Instead, it is ontogenesis, the inherently orderly but contingent coming into being, that expresses what is essential about the emergence of pattern and form without trapping us in infinite cognitive regress (where was the pattern before it got here?). A proper view of ontogeny, however, that doesn't simply resolve into one of the old ones, requires that the idea of ontogenesis apply not only to bodies and minds, but to information, plans and all the other cognitive-causal entities...that supposedly regulate their development. Developmental information itself, in other words, has a developmental history. It neither preexists its operations nor arises from random disorder. It is neither necessary, in an ultimate sense, nor a function of pure chance, though contingency and variation are crucial to its formation and its function. Information is a difference that makes a difference, and what it "does" or what it means is thus dependent on what is already in place and what alternatives are being distinguished.(p. 2-3)"
      (as cause) "The discovery of DNA and its confirmation of a gene theory that had long been in search of its material agent offered an enormously attractive apparent solution to the puzzle of the origin and perpetuation of living form. A material object housed in every part of the organism, the gene semed to bridge the gap betwen inert matter and design; in fact, genetic information, by virtue of the meanings of in- formation as "shaping" and as "animating," promised to supply just the cognitive and causal functions needed to make a heap of chemicals into a being (p. 12)."
      (as cause) "In such information-based treatments of biological processes, then, a transition is quickly made from formal considerations of numbers of alternatives and reduction of uncertainty to the familiar concerns with causal control of development and adaptation. A related transition is that between the restricted analytical use of genetic determination or control as explanation of differences within or across populations (a use that, being based on correlation, is compatible with information theory) and as explanation of kinds of development (maturational, canalized, governed by epigenetic rules) and characters (innate, biological, programmed, phylogenetically derived). This shift from information as a measure of unpredictability to information as an explanation of predictability is frequent and unreflective, and, given the technical legitimacy of terms like "determinant" and "control" in certain contexts, lend a similar legitimacy to their misuse" (p. 67).
      Control cannot be identified with one aspect of a process. It is contingent, causally interdependent processes.
      (as commodity)"In an increasingly technological, computerized world, information is a prime commodity, and when it is used in biological theorizing it is granted a kind of atomistic autonomy as it moves from place to place, is gathered, stored, imprinted and translated. It has a history only insofar as it is accumulated or transferred. Information, the modern source of form, is seen to reside in molecules, cells, tissues, "the environment," often latent but causally potent, allowing these entities to recognize, select and instruct each other, to construct each other and themselves, to regulate, control, induce, direct and determine events of all kinds. When something marvelous happens, whether it be the precise choreography of an "instinctive" behavior or the formation of an embryonic structure, the question is always, Where did the information come from? Was it already in the animal or the developing tissue, or did it have to be put in through learning or perhaps some embryological organizer? Was selection or instruction responsible? Is this a phylogenetic or an ontogenetic adaptation? (Was the information acquired by the species or must it have been acquired through individual experience?) The ease with which extreme nature and nurture positions are parodied ensures that no one will stand behind either straw man. No one really argues, that is, either that livers and ideas are literally in the cell or that organism are devoid of structure, pristine pages on which anything at all may be written (and even a page has structure!). Or. to put it in negative terms, no one contends that either developmental conditions or the genes are totally irrelevant to development. Any locution that dissociates one from the straw man even minimally, however, seems to offer protection from criticism. Encoded potential and biological constraints, then allow everyone to return to work, the pesky conceptual issues behind them, peace apparently restored (p. 1-2).
      (as constituted by the developmental system)" In fact, the habit of thinking about phylogeny and ontogeny as alternative processes whereby information enters the organism is the very frame on which our endless nature- nurture disputations are woven. Nativism and empiricism require each other as do warp and weft. What they share is the belief that information can preexist the processes that give rise to it. Yet information "in the genes" or "in the environment" is not biologically relevant until it participates in phenotypic processes. Once this happens, it becomes meaningful in the organism only as it is constituted by its developmental system. The result is not more information but significant information (p. 13).
      (as constituted by the developmental system) "The significance of any information influencing developmental processes...whether from genetic transcription or from other events, is defined by the developing system itself (p. 66)."
      (contingency of) "Chromosomal form is an interactant in the choreography of ontogeny; the "information" it imparts or the form it influences in the emerging organism depends on what dance is being performed, when, where and with whom (p. 22)."
      ("as difference that makes a difference") "The "informational" significance of any developmental influence, as we have seen, depends on the state of the entire developmental system, including genes, phenotype and relevant aspects of surround, and on the level and the type of analysis. Developmental state is a kind of temporal slice through the life cycle. It carries the evidence of past gene transcriptions, mechanical influences inside and outside the organism, results of past activities, nutrition or the lack of it, and so on, and it has certain prospects for change. If we are guided by the notion of information as the difference that makes a difference, then what developmental interactant makes a difference depends on what is developing, and how. Understanding ontogeny becomes partly a matter of charting the shifts from one source of change (including intraorganismic processes) to another, as one interaction alters the developmental system in a way that provides transition to the next. Equally important are the means whereby stability is achieved. In addition, the organism can be investigated at the behavioral, physiological or other level, and comparisons made between a system and itself at an earlier time, among members of the same species, between a species and its phylogenetic ancestors or its contemporary relatives; again, what makes a difference depends on what question is being asked.
      Some traditional nature-nurture questions can thus be restated in terms of developmental systems. The benefit of this restatement is that it makes clear what is really being asked, and therefore what would constitute an answer. The mischief of the nature-nurture complex was that it conflated questions and answers both, so that an apparently spontaneous change in behavioral patter, for example, was attributed to the genes, and thus concluded to be unchangeable in the individual, to be universal in the species, to show cross-species identity. Conversely, a cross-species resemblance was taken to mean that no learning had taken place, that no learning would have an impact, that the trait was more real than others that did not show such resemblance. To gain information we need to specify a context and a set of possibilities. It is in this sense that organisms generate information (Klopfer, 1969, 1973, p. 27), and it is in much the same sense that scientists do. Events do not carry already existing information about their effects from one place to the next, the way we used to think copies of objects had to travel to our minds for us to perceive them. They are given meaning by what they distinguish. This why a gene has different effects in different tissues and at different times, why a stimulus calls out different responses, including no response, at different times or in different creatures, and why an observation that is meaningless or anomalous at one stage of an investigation or to one person becomes definitive under other circumstances. A difference that makes a difference at one level of analysis, furthermore, may or may not make a difference at another. This is, in fact, the key to understanding apparent spontaneity...(p. 143-144).

      Problems in moving from Information Theory as applied to communications (simple systems) to living systems.

      The "open system."
      The "complex system."


      Ashby, W.R. (1964). An Introduction to Cybernetics. London; Nethuen & Co. Ltd. (Originally published in 1956).

      (on complexity) "Science stands today on something of a divide. For two centuries it has been exploring systems that are either intrinsically simple or that are capable of being analysed into simple components. The fact that such a dogma as "vary the factors one at a time" could be accepted for a century, shows that scientists were largely concerned in investigating such systems as allowed this method; for this method is often fundamentally impossible in the complex systems. Not until Sir Ronald Fisher's work in the 20's, with experiments conducted on agricultural soils, did it become clearly recognised that there are complex systems that just do not allow the varying of only one factor at a time--they are so dynamic and interconnected that the alteration of one factor immediately acts as cause to evoke alterations in others, perhaps in a great many others. Until recently, science tended to evade the study of such systems, focusing its attention on those that were simple and, especially reducible.
      In the study of some systems, however, the complexity could not be wholly evaded. The cerebral cortex of the free-living organism, the ant-hill as a functioning society, and the human economic system were outstanding both in their practical importance and in their intractability by the older methods. So today we see psychoses untreated, societies declining, and the economic systems faltering, the scientist being able to do little more than to appreciate the full complexity of the subject he is studying. But science today is also taking the first steps towards studying "complexity" as a subject in its own right (p. 5)."
      (on complexity and the search for perspectival adequacy) "These facts emphasise an important matter of principle in the study of the very large system. Faced with such a system, the observer must be cautious in referring to "the system", for the term will probably be ambiguous, perhaps highly so. "The system" may refer to the whole system quite apart from any observer to study it - the thing as it is in itself; or it may refer to the set of variables (or states) with which some given observer is concerned. Though the former sounds more imposing philosophically, the practical worker inevitably finds the second more important. Then the second meaning can itself be ambiguous if the particular observer is not specified, for the sytem may be any one of the many sub-machines provided by homomorphism. Why all these meanings should be distinguished is because different sub-machines can have different properties; so that although both sub-machines may be abstracted from the same real "thing", a statement that is true of one may be false of another.
      It follows that there can be no such thing as the (unique) behviour of a very large system, apart from a given observer. For there can legitimately be as many sub-machines as observers, and therefore as many behaviours, which may actually be so different as to be incompatible if they occurred in one sytem...
      The point of view taken here is that science (as represented by the observer's discoveries) is not immediately concerned with discovering what the system "really" is but with co-ordinating the various observers' discoveries, each of which is only a portion, or an aspect, of the whole truth (p. 106-107)."
      (on communication) "Communication thus necessarily demands a set of messages. Not only is this so, but the information carried by a particular message depends on the set it comes from. The information conveyed is not an intrinsic property of the individual message (p. 124)."
      (on coupled systems) "A fundamental property of machines is that they can be coupled. Two or more whole machines can be coupled to form one machine; and any one machine can be regarded as formed by the coupling of its parts, which can themselves be thought of as small, sub-, machines. The coupling is of profound importance in science, for when the experimenter runs an experiment he is coupling himself temporarily to the system that he is studying (p. 48)."
      (on memory and the observer) "If a determinate system is only partly observable, and thereby becomes (for that observer) not predictable, the observer may be able to restore predictability by taking the sytem's past history into account, i.e. by assuming the existence within it of some form of "memory".(p. 115>
      "Thus the possession of "memory" is not a wholly objective property of a system - it is a relation between a system and an observer; and the property will alter with variations in the channel of communication between them...Thus to invoke "memory" in a system as an explanation of its behaviour is equivalent to declaring that one cannot observe the system completely. (p. 116)."
      "Thus, suppose I am in a friend's house and, as a car goes past outside, his dog rushes to a corner of the room and cringes. To me the behaviour is causeless and inexplicable. Then my friend says, "He was run over by a car six months ago." The behaviour is now accounted for by reference to an event of six months ago. If we say that the dog shows "memory" we refer to much the same fact - that his behaviour can be explained, not by reference to his state now but to what his state was six months ago. If one is not careful one says that the dog "has" memory, and then thinks of the dog as having something, as he might have a patch of black hair. One may then be tempted to start looking for the thing; and one may discover that this "thing" has some very curious properties.
      Clearly, "memory" is not an objective something that a system either does or does not possess; it is a concept that the observer invokes to fill in the gap caused when part of the system in unobservable. The fewer the observable variables, the more will the observer be forced to regard events of the past as playing a part in the system's behaviour. Thus "memory" in the brain is only partly objective. No wonder its properties have sometimes been found to be unusual or even paradoxical. Clearly the subject requires thorough re-examination from first principles (p. 117)."
      (on variety and partitioning) "It will be noticed that a set's variety is not an intrinsic property of the set: the observer and his powers of discrimination may have to be specified if the variety is to be well defined (p. 125)."
      The issue of partitioning is a developmental issue.


      Artificial Intelligence

      Throughout our history we seem to find it useful to model ourselves after our latest inventions. The bellows became the model of how our lungs worked; the machine for the human body, the telephone switchboard for our brain; and today, we model ourselves after the computer. The cognitive approach seeks to understand the codes that program our minds/brains and which account for the orderliness of behavior.

      However, living systems are not computers. They do not function in a computational manner. Our behavior, that is, cannot be explained by the application of algorithms.

      A good discussion of the differences between organisms and computational machines is to be found in Dreyfus, H.L. (1979). What Computers Can't Do. N.Y.: Harper and Row. This book gives a number of strong arguments against the possibility of creating machines to replicate human behavior. However, whether computers can eventually do what humans can do is not important.


      Webb, B. (1996 December). A Cricket Robot. Scientific American, vol. 275(6).

      This article is interesting from the point of view of artificial intelligence (i.e. the replication of behavior with robots) and as a fruitful method for modeling the behavior of organisms.

      The author points out the difficulties with modeling even the most apparently simple, automatic behaviors of organisms. The interaction of sensors and actuators with the environment is always complicated and requires complicated sets of algorithms to achieve the desired result, if it can be achieved at all (see also Dreyfus, above, for a discussion of this problem). The approach usually taken is to try to handle complex environmental circumstances with equally sophisticated algorithms. In other words, the environment is considered as something to be overcome by a central processing unit which calculates the most appropriate response to an infinite number of possible situations.

      Her goal is stated as follows:

      "...to design the robot so that its interaction with the environment is exploited rather than resisted. For example, instead of attempting to force the robot to travel a straight- line course, it could be programmed to follow contours of the terrain that lead to its destination - circumventing rather than conquering hills in its path. Through this type of approach, what seems like complex behavior in a robot can come from a surprisingly uncomplicated control algorithm."

      Her robot was designed with the intention of modeling the behavior of a female cricket in moving toward the sound made by a possible mate. This is a quite robust behavior in the female cricket.

      The approach to the problem consisted of three aspects:

      • Analysis of the stimulus. The sound properties of the male cricket's chirping were analyzed.
      • Analysis and construction of cricket morphology. The robot was constructed in accordance with what is known about cricket physical and neurological functioning.
      • Writing of the code to program the robot. Since the robot was designed to respond to a narrow range of stimuli in a manner which presumably modeled the cricket's morphology, the amount of code required was limited to about 100 lines - a surprisingly small amount.

      The amount of "information processing" required by the robot was minimal since it's morphology only allowed it to respond to just any stimulus. The morphology was the first phase in information processing. It could not respond to the wrong cricket song.

      "The evidence from the robot suggests that situational factors, rather than additional neural processing mechanisms, may explain the effects of chirp structure on the female's movement toward a potential mate."
      "Because I programmed the robot, I knew it was not capable of distinguishing or deciding between the sounds. Yet again it appears that it is the interaction of the robot's uncomplicated mechanisms with particular sound fields that produces this interesting - and useful - behavior."
      "More generally, it shows that a rather competent and complex performance can come from a simple control mechanism, provided it interacts in the right way with its environment."

      This research suggests that artificial intelligence experts, and cognitive psychologists, may be attempting to place too much central control in the "brain". Instead, the boundaries between organism and environment are blurred, and control is more evenly distributed between the two. In this case the investigator has been able to specify the relevant characteristics of a particular environmental stimulus and relevant characteristics of the organism's morphology. The "code" necessary for programming the microchip we may consider represents the unknown factors of the cricket's neurophysiology, the "black box." It is the stuff we fill in when part of the system is unobservable, and in this case it takes the form it does because of the requirements of constructing a functioning machine in contrast to the organism. Most importantly, by looking at the code itself I am sure that we could not understand how the machine does its thing. The behavior of the machine is not to be found in any simple location - not in the machine, simply because we constructed it, nor in the nature of the environmental stimulus.





Back to top
Back to Roadmarks Main Page
Back to Home Page


E-Mail: Laurence E. Heglar, PhD