Beta Phase: Square45 is currently in beta testing. Expect some features or content to be incomplete or missing.
45

Claude Shannon's Information Theory

A framework for quantifying information, providing the basis for secure communication protocols and cryptographic design.
📜

The statement of the theorem

Let XX and YY be discrete random variables with joint probability mass function P(x,y)P(x, y). The entropy of XX is defined as H(X)=−∑xP(x)log⁡2P(x)H(X) = -\sum_{x} P(x) \log_2 P(x). The joint entropy is H(X,Y)=−∑x,yP(x,y)log⁡2P(x,y)H(X, Y) = -\sum_{x, y} P(x, y) \log_2 P(x, y). The mutual information I(X;Y)I(X; Y) quantifies the reduction in uncertainty about XX given YY, and is formally defined as:\nI(X;Y)=∑x,yP(x,y)log⁡2(P(x,y)P(x)P(y))I(X; Y) = \sum_{x, y} P(x, y) \log_2 \left(\frac{P(x, y)}{P(x)P(y)}\right) \nIt satisfies the chain rule: H(X,Y)=H(X)+H(Y∣X)H(X, Y) = H(X) + H(Y|X), where H(Y∣X)=−∑x,yP(x,y)log⁡2P(y∣x)H(Y|X) = -\sum_{x, y} P(x, y) \log_2 P(y|x) is the conditional entropy.