This module is designed to introduce you to the main concepts of information theory (i.e., the theory of information and data). Information theory provides you with the metrics (e.g., entropy of a source, mutual information) and with the tools to estimate the best achievable performance of a communication systems. It also allows you to predict the performance of a compression algorithm (eg., image compression algorithm). For this module we require a background knowledge of mathematics and elementary probability.

At the end of the module, you are expected to be able to (1) apply the concept and properties of entropy and mutual information to design communication systems, (2) prove the source coding theorem, (3) deduce the fundamental performance limits of noisy communication channels, (4) predict the performance of a compression algorithm by using the rate-distortion function, (5) apply network information theory to design complex communication systems.

Elements of information theory of discrete systems; information measures, memoryless and memory sources, the noiseless coding theorem; Methods of source coding; Information theory of continuous systems; Shannon's capapcity theorem and its interpretation; Lossy source coding and rate-distortion theory; Network information theory.