Information Theory

Course content

The lecture provides an introduction to the fundamentals of Shannon information theory. The goal is that students can derive the main information theoretic results on maximal achievable lossless (source coding) and lossy (rate distortion theory) compression of data and on maximum data rates for reliable data transmission (channel coding). The methods and tools required, e.g., information measures (entropy, mutual information, capacity etc.) and their properties (typical sequences) will be covered as well as practical applicable simple codes (block, turbo and polar codes).

Content:

• Basics from probability theory
• Event, probability, random variable, random vector, stochastic process, convergence of random series, convergence theorems
• Basics from information theory
• Measures for discrete random varaibles: entropy, conditional entropy, relativ entropy, mutual information, conditional mutual information, inequalities
• Measures for continous random variables: differential entropy, conditional differential entropy, relative entropy, mutual information, inequalities
• Measure for random series
• Typical sequences and asymptotic equipartition property
• Source and source coding
• Definition and properties
• Source coding for discrete memoryless sources (fixed and variable-length)
• Selected source codes: Morse, Huffman, Shannon-Fano-Elias
• Data transmission and channel capacity
• Discrete memoryless channel: channel coding theorem
• Discrete memoryless channel with state: channel capacities
• Gaussian channel: model and channel coding theorem
• Bandlimited Gaussian channel, vector valued channels