Introduction to Information Theory

 

The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.
  (Claude Shannon, 1948)

In the first part of this book we are going to study how to measure information content; we are going to learn by how much data from a given source can be compressed; we are going to learn how, practically, to achieve data compression; and we are going to learn about how to communicate perfectly over an imperfect communication channel.

We start by getting a feeling for this last problem.

 
• How can one achieve perfect communication over an imperfect, noisy commmunication channel?
  • The physical solution
  • The `system' solution
• Error correcting codes for the binary symmetric channel
  • Repetition codes
  • Block codes -- the (7,4) Hamming code
    • Decoding the (7,4) Hamming code
  • Pictorial view of the Hamming code
    • Encoding
    • Decoding
  • Summary of Hamming code properties
  • Summary of codes' performances
• What performance could the best codes achieve?
  • Example: f=0.1
• Summary
• Homework
• Probability and entropy -- definitions and notation
  • Probabilities and ensembles
  • Meaning of probability.
  • Definition of Entropy and related functions
  • Other useful definitions
• Exercises