Current Slide

Brief Review of Entropy

• Entropy (Information Theory)
• A measure of uncertainity associated with a random variable
• Calculation: for a discrete random variable Y taking m distinct values {y1,...,ym},
  \[H(Y)=-\sum_{i=1}^{m}p_{i}log(p_{i}), p_{i}=P(Y=y_{i})\]
  

• Interpretation:
  • Higher entropy=>higher uncertainty
    
  • Lower entropy=>lower uncertainty
• Conditional entropy
  \[H(Y|X)=\sum_{x}p(x)H(Y|X=x)\]