Information Theory

import math
import numpy
print (math.log(1.0/0.98)) # Natural log (ln)
print (numpy.log(1.0/0.02)) # Natural log (ln)
print (math.log10(1.0/0.98)) # Common log (base 10)
print (numpy.log10(1.0/0.02)) # Common log (base 10)
print (math.log2(1.0/0.98)) # Binary log (base 2)
print (numpy.log2(1.0/0.02)) # Binary log (base 2)

1. First law $\log A + \log B = \log AB$ Compute the following expression:$\log_{10}3 + \log_{10}4$2. Second law$\log A^n = n\log A$Compute the following expression:$\log_24^6$3. Third law $\log A − \log B = \log \frac{A}{B} .$ Therefore, subtracting $\log B$ from in $\log A$ in $\log \frac{A}{B}$ . Compute the following expression: $\log_e15 - \log_e3$

1. For an event which is certain to happen, what is the entropy? (a) 1.0 (b) 0.0 (c) The entropy is undefined (d) −1 (e) 0.5 (f) $log_2(N)$, N being the number of possible events
2. For N equiprobable events, what is the entropy? (a) 1.0 (b) 0.0 (c) The entropy is undefined (d) −1 (e) 0.5 (f) $log_2(N)$

Enumerate these properties.

Mark the correct answer: Information is a/an [decrease/increase] in uncertainty.

1. Define what it actually meant by being surprised.
2. Describe how it is related to the likelihood of an event happening.
3. True or False: The less likely the occurrence of an event, the smaller information it conveys.

1. Assume a source that provides a constant stream of N equally likely symbols ${x1, x2, . . . , xN }$. What does Shannon's formulae reduce to in this particular case?
2. Assume that each equiprobable pixel in a monochrome image that is fed to a DL classification pipeline, can have values ranging from 0 to 255. Find the entropy in bits.

$H = -\sum_{a=1}^NP_a\log_2P_a \\(bits per symbol)$

1. Plot a graph of the curve of probability vs. uncertainty.
2. Complete the sentence: The curve is [symmetrical/asymmetrical].
3. Complete the sentence: The curve rises to a [minimum/maximum] when the two symbols are equally likely ($P_a = 0.5$).

$H(X, Y ) = H(X, Z)$

1. Complete the sentence: We can predict ‘heads’ for each flip with an accuracy of [___]%.
2. Complete the sentence : If the result of the coin toss is ‘heads’,the amount of Shannon information gained is [___] bits.
3. Complete the sentence: If the result of the coin toss is ‘tails’, the amount of Shannon information gained is [___] bits.
4. Complete the sentence : It is always true that the more information is associated with an outcome, the [more/less] surprising it is.
5. Provided that the ratio of tosses resulting in ‘heads’ is $p(x_h)$, and the ratio of tosses resulting in ‘tails’ is$p(x_t)$, and also provided that$p(x_h)+p(x_t) = 1$, what is formulae for the average surprise?
6. What is the value of the average surprise in bits?

density functions P and Q.

$D_{KL}(P||Q) != D_{KL}(Q||P)$ 2. True or False: The KL-divergence satisfies the triangle inequality. 3. True or False: The KL-divergence is not a distance metric. 4. True or False: In information theory, KLD is regarded as a measure of the information gained when probability distribution Q is used to approximate a true probability distribution P . 5. True or False: The units of KL-divergence are units of information. 6. True or False: The KLD is always non-negative, namely: $D_{KL}(P||Q) ≥ 0.$ 7. True or False: In a decision tree, high information gain indicates that adding a split to the decision tree results in a less accurate model.

e.g.: $\text{expansion}=+, \text{no expansion}=-$ Referring to table (4.3), each row indicates the observed values, columns $(θi)$ denote features and class label $(γ)$ denotes whether expansion was observed.

Referring to table (4.4), each row indicates the observed values, columns $(θ_i)$ denote features and class label $(γ)$ denotes whether shrinkage was observed.

$I(X; Z) := H(X) − H(X | Z)$ where H is the Shannon entropy, and $H (X | Z )$ is the conditional entropy of $Z$ given $X$.

2. Define the term convex function.
3. State Jensen's inequality and its implications.

is always greater or equal to zero.