Why is Shannon Entropy of individual sections always between 0-8. Also why we need to create a 256 freq array while calculating the Shannon Entropy?
2
-
The mathematical answer is here: stats.stackexchange.com/questions/95261/…– DavidDec 20, 2020 at 22:55
-
An intuitive answer is here, which also answers your question about the frequency array (hint: each byte represents one of 256 discrete states): towardsdatascience.com/…– DavidDec 20, 2020 at 22:59
Add a comment
|
1 Answer
The channel capacity of a single byte samples has a maximum of 8 bits.
Another way of thinking about it: If a single byte has the same value for every sample, you need 0 additional bits of information to describe the values.
If a single byte takes on 256 different values, [0 to 255], then for every sample you will need 8 additional bits of information to uniquely describe the values.
For example, if you were measuring the Shannon Entropy of a collection of Short values (2 bytes / 16 bits) it would range from 0 (constant) to 16 (completely random).
