Course Co-ordinated by IIT Kanpur
 Coordinators IIT Kanpur

Untitled Document

Information Theory answers two fundamental questions: what is the maximum data rate at which we can transmit over a communication link, and what is the fundamental limit of data compression. In this course we will explore answers to these two questions. We will study some practice source compression algorithms. We will also study how to compute channel capacity of simple channels.

 Week Topics 1. Introduction: Entropy, Relative Entropy, Mutual Information;  Information Inequalities;  Block to variable length coding-I: Kraft’s inequality. 2. Block to variable length coding-II: Huffman coding;  Variable to block length coding: Tunstall coding. 3. Block to block length coding: Typical sequences;  Variable to variable length coding-I: Arithmetic codes;  Variable to variable length coding-II: Lempel-Ziv codes. 4. Asymptotic Equipartition Property;  Coding for sources with memory. 5. Noisy channel coding theorem;  Converse of noisy channel coding theorem;  Channel capacity of discrete memoryless channels. 6. Differential entropy ;  Gaussian Channel;  Parallel Gaussian Channel. 7. Rate Distortion Theory;  Blahut-Arimoto Algorithm for computation of channel capacity and rate- distortion function. 8. An introduction to Network Information Theory-I;  An introduction to Network Information Theory-II.

Basic knowledge of probability theory and digital communications

• James L. Massey, Lecture notes on ``Applied Digital Information Theory I''.
• Thomas M. Cover, Joy A. Thomas, ``Elements of Information Theory'', 2nd Edition, John Wiley & Sons, 2006.
• Robert G. Gallager, ``Information Theory and Reliable Communications'', John Wiley & Sons, 1968.
• Raymond W. Yeung, ``Information Theory and Network Coding'', Springer, 2008.
• David J. C. MacKay, ``Information Theory, Inference, and Learning Algorithms'', Cambridge University Press.
• Robert Ash, ``Information Theory'', Dover Publications, 1965.
• Imre Csiszar and Jonos Korner, ``Information Theory'', Second edition, Cambridge University Press, 2011.