A Short Course in Information Theory
8 lectures by David J.C. MacKay
January 1995. Cavendish Laboratory, Cambridge, Great Britain.
Summary
Is it possible to communicate reliably from one point
to another if we only have a noisy communication
channel? How can the information content of a random
variable be measured? This course will discuss the
remarkable theorems of Claude Shannon, starting from
the source coding theorem, which motivates the entropy
as the measure of information, and culminating in the
noisy channel coding theorem. Along the way we will
study simple examples of codes for data compression
and error correction.
This will be an informal course. All are welcome to
attend. The level of presentation is intended to be
appropriate for graduate students and final year
undergraduates.
You might also be interested in
my book
on Information Theory, Inference and Learning Algorithms (640 pages long, published by C.U.P. Sept 2003, and available online),
which grew out of this short course.
The postscript files can be obtained not only from my UK web server but also from a
mirror in North America (Toronto). Please click appropriately.

Course outline (postscript, 1 page).
 ps mirror 
pdf
 pdf mirror 

Lecture 1 notes (postscript, 2 pages).
 ps mirror 
pdf
 pdf mirror 
 Definitions of Probabilities and Entropies.
These notes do not include the main part of lecture 1, viz, the
45 minute overview of the noisy channel coding theorem.
(You can find that in the first chapter of my book)

Lecture 2 notes (postscript, 3 pages).
 ps mirror 
pdf
 pdf mirror 
 Why is entropy a fundamental measure of information content?
Assymptotic equipartition and the source coding theorem.
Note: there is a figure on page 1 of this document
which does not appear under
ghostview for some reason.

Lecture 3 notes (postscript, 3 pages).
 ps mirror 
pdf
 pdf mirror 
 Data compression I: Symbol codes

Lecture 4 notes (postscript, 2 pages).
 ps mirror 
pdf
 pdf mirror 
 Data compression II: Arithmetic coding

Lecture 5 notes (postscript, 4 pages).
 ps mirror 
pdf
 pdf mirror 
 Noisy Channel Coding Theorem I

Lecture 6 notes (postscript, 2 pages).
 ps mirror 
pdf
 pdf mirror 
 Noisy Channel Coding Theorem II

Lectures 7 and 8 notes (postscript, 4 pages).
 ps mirror 
pdf
 pdf mirror 
 Error Correcting Codes and Real Channels.
In lecture 7 I also described work on
Decoding by variational free energy minimization.

Additional notes (postscript, 3 pages).
 ps mirror 
pdf
 pdf mirror 
 Bayesian inference (notes prepared for a lecture that didn't happen)
 This site has received the LookSmart Editor's Choice Award
and is listed in the Infront Directory.
David MacKay / mackay@mrao.cam.ac.uk 
home page.