Claude Shannon's creation in the 1940's of the subject of information
theory is arguably one of the great intellectual achievements of the
twentieth century. Information theory has had an important and
significant influence on mathematics, particularly on probability
theory and ergodic theory, and Shannon's mathematics is in its own
right a considerable and profound contribution to pure mathematics.
But Shannon did his work primarily in the context of communication
engineering, and it is in this area that it stands as a unique
monument. In his classical paper of 1948 and its sequels, he
formulated a model of a communication system that is distinctive for
its generality as well as for its amenability to mathematical
analysis. He formulated the central problems of theoretical interest,
and gave a brilliant and elegant solution to these problems. We
preface this section of his collected works with a very short
description of this pioneering work.
Let us look first at his model. Shannon saw the communication process as essentially stochastic in nature. The semantic meaning of information plays no role in the theory. In the Shannon paradigm, information from a "source" (defined as a stochastic process) must be transmitted though a "channel" (defined by a transition probability law relating the channel output to the input). The system designer is allowed to place a device called an "encoder" between the source and channel which can introduce a fixed though finite (coding) delay. A "decoder" can be placed at the output of the channel. The theory seeks to answer questions such as how rapidly or reliably can the information from the source be transmitted over the channel, when one is allowed to optimize with respect to the encoder/decoder?
Shannon gives elegant answers to such questions. His solution has two parts. First he gives a fundamental limit which, for example, might say that for a given source and channel, it is impossible to achieve a fidelity or reliability level better than a certain value. Second, he shows that for large encoder delays, it is possible to achieve performance that is essentially as good as the fundamental limit. To do this, the encoder might have to make use of a complex code which is not necessarily implementable in practice.
One of Shannon's most brilliant insights was the separation of problems like these (where the encoder must take both the source and channel into account) into two coding problems. He showed that with no loss of generality one can study the source and channel separately and assume that they are connected by a digital (say binary) interface. One then finds the (source) encoder/decoder to optimize the source-to-digital performance, and the (channel) encoder/decoder to optimize the performance of the channel as a transmitter of digital data. Solution of the source and channel problems leads immediately to the solution to the original joint source-channel problem. The fact that a digital interface between the source and channel is essentially optimal has profound implications in the modern era of digital storage and communication of all types of information.
Thus the revolutionary elements of Shannon's contribution were the invention of the source-encoder-channel-decoder-destination model, and the elegant and remarkably general solution of the fundamental problems which he was able to pose in terms of this model. Particularly significant is the demonstration of the power of coding with delay in a communication system, the separation of the source and channel coding problems, and the establishment of fundamental natural limits on communication.
In the course of developing the solutions to the basic communication problem outlined above, Shannon introduced several mathematical concepts. Primary among these is the notion of the "entropy" of a random variable (and by extension of a random sequence), the "mutual information" between two random variables or sequences, and a calculus that relates these quantities and their derivatives. He also achieved spectacular success with his technique of random coding, in which he showed that an encoder chosen at random from the universe of possible encoders will, with high probability, give essentially optimal performance.
Shannon's work, as well as that of his legion of disciples, provides a crucial "knowledge base" for the discipline of communication engineering. The communication model is general enough so that the fundamental limits and general intuition provided by Shannon theory provide an extremely useful "roadmap" to designers of communication and information storage systems. For example, the theory tells us that English text is not compressible to fewer than about 1.5 binary digit per English letter, no matter how complex and clever the encoder/decoder. Most significant is the fact that Shannon's theory showed how to design more efficient communication and storage systems by demonstrating the enormous gains achievable by coding, and by providing the intuition for the correct design of coding systems. The sophisticated coding schemes used in systems as diverse as "deep-space" communication systems (for example NASA's planetary probes), and home compact disk audio systems, owe a great deal to the insight provided by Shannon theory. As time goes on, and our ability to implement more and more complex processors increases, the information theoretic concepts introduced by Shannon become correspondingly more relevant to day-to-day communications.