Information Theory

(3-0-0-3)

CMPE Degree: This course is Not Applicable for the CMPE degree.

EE Degree: This course is Not Applicable for the EE degree.

Lab Hours: 0 supervised lab hours and 0 unsupervised lab hours.

Technical Interest Group(s) / Course Type(s): Telecommunications

Course Coordinator:

Prerequisites: ECE 3075

Corequisites: None.

Catalog Description

To introduce the mathematical theory of communications. Emphasis will be
placed on Shannon's theorems and their use in the analysis and design of
communication systems

Textbook(s)

Elements of Information Theory

Course Outcomes

Not Applicable

Student Outcomes

In the parentheses for each Student Outcome:
"P" for primary indicates the outcome is a major focus of the entire course.
“M” for moderate indicates the outcome is the focus of at least one component of the course, but not majority of course material.
“LN” for “little to none” indicates that the course does not contribute significantly to this outcome.

1. ( Not Applicable ) An ability to identify, formulate, and solve complex engineering problems by applying principles of engineering, science, and mathematics

2. ( Not Applicable ) An ability to apply engineering design to produce solutions that meet specified needs with consideration of public health, safety, and welfare, as well as global, cultural, social, environmental, and economic factors

3. ( Not Applicable ) An ability to communicate effectively with a range of audiences

4. ( Not Applicable ) An ability to recognize ethical and professional responsibilities in engineering situations and make informed judgments, which must consider the impact of engineering solutions in global, economic, environmental, and societal contexts

5. ( Not Applicable ) An ability to function effectively on a team whose members together provide leadership, create a collaborative and inclusive environment, establish goals, plan tasks, and meet objectives

6. ( Not Applicable ) An ability to develop and conduct appropriate experimentation, analyze and interpret data, and use engineering judgment to draw conclusions

7. ( Not Applicable ) An ability to acquire and apply new knowledge as needed, using appropriate learning strategies.

Strategic Performance Indicators (SPIs)

Not Applicable

Course Objectives

Topical Outline

Entropy and Mutual Information Theory
-Joint Entropy, Conditional Entropy
-Data Processing Theorem
-Fano's Inequality

Asymptotic Equipartition Principle
-Typical Sequences
-Entropy, Source Coding and the AEP
-Joint Typicality (Neuhoff/Forney notes)

Entropy Rate
-Conditional Independence and Markov Chains
-Entropy Rate

Lossless Source Coding
-Kraft Inequality
-Shannon and Huffman Codes
-Shannon, Fano, Elias Codes
-Arithmetic Codes
-Lempel Ziv Codes

Channel Capacity
-Symmetric Channels
-Discrete Memoryless Channels and Their Capacity
-Arimoto-Blahut Algorithm
-Proof of the Channel Coding Theorem
-Converse of Channel Coding Theorem

Differential Entropy
-Entropy, Mutual Information, AEP for Continuous rv's

Gaussian Channel
-Capacity of AWGN, Bandlimited AWGN Channels
-Capacity of Nonwhite Channels: Water Filling

Rate Distortion Theory
-Quantization
-Rate Distortion Functions
-Vector Quantization
-Vector Quantization Gains
-Vector Quantization Design

Multiuser Information Theory (as time allows)

Information Theory and Statistics (as time allows)