When I first read Shannon’s 1948 paper “A Mathematical Theory of Communication”, one set of numbers particularly jumped out at me: the numbers for the difference between the error rate of a communications channel and the rate at which information is lost from it. I recently ran into a free link to the paper (in a blog post by Federico Pereiro), which reminded me of it. As Shannon explains the issue:

Suppose there are two possible symbols 0 and 1, and we are transmitting at a rate of 1000 symbols per second with probabilities p0 = p1 = 1/2. Thus our source is producing information at the rate of 1000 bits per second. During transmission the noise introduces errors so that, on the average, 1 in 100 is received incorrectly (a 0 as 1, or 1 as 0). What is the rate of transmission of information? Certainly less than 1000 bits per second since about 1% of the received symbols are incorrect. Our first impulse might be to say the rate is 990 bits per second, merely subtracting the expected number of errors. This is not satisfactory since it fails to take into account the recipient’s lack of knowledge of where the errors occur. We may carry it to an extreme case and suppose the noise so great that the received symbols are entirely independent of the transmitted symbols. The probability of receiving 1 is 1/2 whatever was transmitted and similarly for 0. Then about half of the received symbols are correct due to chance alone, and we would be giving the system credit for transmitting 500 bits per second while actually no information is being transmitted at all. Equally “good” transmission would be obtained by dispensing with the channel entirely and flipping a coin at the receiving point.

He then goes on to calculate the transmission rate, for this case, as being 919 bits per second. That is, if 1% of the bits are flipped, it results in about an 8% loss in the capacity of the channel. In other words, to transmit data correctly in the presence of a 1% error rate, you have to surround it with error-correction codes that bulk it up by about 8% – in practice, by even more than that, since error-correction schemes are not perfectly efficient.

What is interesting is to consider the implications of this in less formal settings: in news reporting, for instance. A 1% error rate would be very good for a newspaper; even the best magazines, with fact checkers who call around to check details of the articles they publish, can only dream of an error rate as low as 1%. Most of their errors are things that only specialists notice, but they are still errors, and often ones which significantly change the moral of the story; specialists frequently sigh at the inaccuracy of the news coverage of their specialty. But that unattainable 1% error rate would still mean that, at best, one would have to throw out 8% of what is said as being unreliable. That is, if one were interested in perfect truth. People in the news business have long since become inured to the fact that what they print is imperfect, and their main concern is that the errors they make not be of the laughable sort – or at least not laughable to the general population. But if one wants to figure out the world, that is not good enough.

To make things worse, most of the world does not encode information in error-correction schemes to cover for the press’s errors. When anyone does so (and political groups have learned to do so), the scheme they usually use is simple repetition: saying things multiple times, in varying language. That’s quite an inefficient error detection/correction scheme, but is the only one that most recipients can be expected to decode and that can get through a human news system in the first place: people would look at you quite funny if you tried to speak in Hamming codes.

For the rest of the information in the world – the stuff not sponsored by any political group – the situation is even worse. In the absence of deliberately added redundancy, one has to use whatever inherent redundancy there is in the information, and mistrust the stuff that isn’t confirmed that way, which is typically more than 90% of it.

This is why reliable sources of information are so valuable: one doesn’t have to regard them with near-fanatical mistrust. It is tempting, once one knows the truth about something, to be liberal in excusing errors in descriptions of it, on the grounds that those descriptions are mostly right. But that is to forget, as Shannon puts it, “the recipient’s lack of knowledge of where the errors occur”, for recipients who are still struggling to find the truth. It may be right to excuse the people who commit those errors, as human beings who perhaps did the best that could be expected of them in the circumstances, but that doesn’t make it right to excuse their work product.