List of Figures
1 Introduction and Preview
1.1Preview of the book
2 Entropy,Relative Entropy and Mutual Information
2.1 Entropy
2.2 Joint entropy and conditional entropy
2.3 Relation entropy and mutual information
2.4 Relationship between entropy ad mutual information
2.5 Chain rules for entropy,relative entropy and mutual information
2.6 Jensen's inequality and its consequences
2.7 The log sum inequality and ist applications
2.8 Data processing inequality
2.9 The second law of thermodymamics
2.10 Sufficient statistics
2.11 Fano;s inequality
Summary of Chapter
Problems for Chatpter
Historical notes
3 The Asymptotic Equipartition Property
3.1 The AEP
3.2 Consequences of the AEP:data compression
3.3 Hign prbability sets and the typical set
Summary of Chapter 3
Problems for Chapter 3
Historical notes
4 Entropy Rates of a Stochastic Process
4.1 Markov chains
4.2 Entropy rate
4.3 Examply:Entropy rateof a random walk on a weighted graph
4.4 Hidden Markov models
Summary of Chapter 4
Problems for Chapter 4
Historical notes
5 Data Compression
5.1 Examples of codes
5.2 Kraft inequality
5.3 Optimal codes
5.4 Bounds on the optimal codelength
5.5 Kraft inequality for uniquely decodable codes
5.6 Huffman codes
5.7 Some comments on Huffman codes
5.8 Optimality of Huffman codes
5.9 Shannon-Fano-Elias coding
5.10 Arithmetic coding
5.11 Competitive optimality of the Shannon code
5.12 Generation of discrete distributions from far coins
Summary of Chapter 5
Problems for Chapter 5
Historical notes
6 Gambling and Data Compression
6.1 The horse race
6.2 Gambling and side information
6.3 Dependent horse races and entropy rate
6.4 The entropy of English
6.5 Data compression and gambling
6.6 Gambling estimate of the entropy of English
Summary of Chapter 6
Problems for Chapter 6
Historical notes
7 Komogorov Complexity
7.1 Models of cmputation
7.2 Kolmogorov complexity:definitions and examples
7.3 Kolmogorov complexity and entropy
7.4 Kolmogorov complexity of integers
7.5 Algorithmically random and incompressible sequences
7.6 Unversal probability
7.7 The halting progblem and the non-computability of Kolmogorov complexity
7.8 Ω/164
7.9 Universal gambling
7.10 Occam's razor
7.11 Kolmogorov complexity and universal probability
7.12 The Dolmogorov sufficeient statistic
Srmmary of Chapter
Problems of Chapter 7
Historical notes
8 Channel Capacity
8.1 Examples of channel capacity
8.2 Symetric channels
8.3 Properties of channel capacity
8.4 Preview of the channel coding theorem
8.5 Definitions
8.6 Jointly typical sequences
8.7 The channle coding theorem
8.8 Zero-error codes
8.9 Fano'inequality and the converse tothe coding theorem
8.10 Equality in thd converse to the channel coding theorem
8.11 Hamming codes
8.12 Feedback capacity
8.13 The joint source channel coding theorem
Summary of Chapter 8
Problems for Chapter 8
Historical notes
9 Differential Entropy
9.1 Definitions
9.2 The AEP for continuous random variables
9.3 Relation of differential entropy to discrete entropy
9.4 Joint and conditional differential entropy
9.5 Relative entroopy and mutual information
9.6 Propertise of differential entropy,relative entropy and mutual information
9.7 Differential entropy bound on discrete entropy
Summary of Chapter 9
Problems for Chapter 9
Historical notes
10 The Gaussian Channel
10.1 The Gaussian channel:definitions
10.2 Converse to the coding theorem for Gaussian channels
10.3 Band-limited channels
10.4 Parallel Gaussian channels
10.5 Channels with colored Gaussian noise
10.6 Gaussian channels with feedback
Summary of Chapter 10
Problems for Chapter 10
Historcal notes
11 Maximum Entropy and Spectral Estimation
11.1 Maximum entropy distributions
11.2 Examples
11.3 An anomalous maximum entropy problem
11.4 Spectrum estimation
11.5 Entropy rates of a Gaussian process
11.6 Burg's maximum entropy theorem
Summary of Chapter 11
Problem for Chapter 11
Historical notes
12 Information Theory and Statistics
12.1 The method of types
12.2 The law of large numbers
12.3 Unuversal source coding
12.4 Large deviation theory
12.5 Examples of Sanov's theorem
12.6 The conditional limit theorem
12.7 Hypothesis testing
12.8 Stein' lemma
12.9 Chernoff bound
12.10 Lempel-Ziv coding
12.11 Fisher information and the Cramer-Rao
inequality
Summary of Chapter 12
Problems for Chapter 12
Historical notes
13 Rate Distortion Theory
13.1 Quantization
13.2 Definitions
13.3 Calculation of the rate distortion function
13.4 Converse to the rate distortion theorem
13.5 Achievability of the rate distortion function
13.6 Strongly typical sequences and rate distortion
13.7 Characterization of the rate distortion function
13.8 Computatio of chammel capacity and the rate distortion function
Summary of Chapter 13
Probems for Chapter 13
Historical notes
14 Network Information Theory
14.1 Gaussian multiple user channels
14.2 Jointly typical sequences
14.3 The multiple access channel
14.4 Encoding of crrelated sources
14.5 Duality between Slepian-Wolf encoding and multiple access channels
14.6 The broadcast channel
14.7 The relay channel
14.8 Source coding with side information
14.9 Rate distortion with side information
14.10 General multiterminal networks
Summary of Chapter 14
Problems for Chapter 14
Historical notes
15 Information Theory and the Stock Market
15.1 The stock market:sone definitions
15.2 Kuhn-Tucker characterizxation of the log-optimal potrfolio
15.3 Asymptotic optimality of the log-optimal porfolio
15.4 Side information and the doubling rate
15.5 Investment in stationary markets
15.6 Competitive optimality of the log-optimal protfolio
15.7 The Shannon-McMillan-Breiman theorem
Summary of Charpter 15
Problems for Charpter 15
Historical notes
16 Inequalities in Theory
16.1 Basic inequalities of information theory
16.2 Differential etropy
16.3 Bounds on entropy and relative entropy
16.4 Inequalities for types
16.5 Entropy rates of subsets
16.6 Entropy and Fisher information
16.7 The entropy power inequality and the BrunnMinkowski inequality
16.8 Inequalites for determinants
16.9 Inequalites for ratios of determinants
Overall Summary
Problems for Chapter
Historical notes
Bibliography
List of Symbols
Index
鄂公网安备 42010302001612号 