# Calculating the entropy of a discrete memoryless source?

33 views (last 30 days)
Daniel_DG on 25 May 2015
Answered: rakesh ranjan on 17 Nov 2020
How could I calculate the entropy of a discrete memoryless source S={a,b,c} in unit of bits? What would be the matlab script for calculating this?

Vinod Sudheesh on 27 May 2015
Hello Daniel,
You could write the script for the same as follows
1) Create a function, say "esingle" that computes the "entropy" of a symbol in the source alphabet. The definition of the function "esingle" could be as follows
function Y=esingle(p)
if(p==0)
Y=0;
return;
else
Y=p*log2(1/p);
end
end
2) Now invoke this function "esingle" using the syntax below to calculate the "entropy" of the source alphabet "a".
a=[0 1 1];
E=sum(arrayfun(@esingle,a/sum(a)));
Hope this helps!
Thanks Vinod
Daniel_DG on 27 May 2015
Thank you very much Vinod. It helped me a lot.

Walter Roberson on 27 May 2015
A source S={a,b,c} is a source with three alphabetic symbols of no particular numeric value. If we know the generating equations for S then we analyze it analytically to determine the entropy. Otherwise the best we can do is estimate the entropy from a stream of the generated symbols. If we have assigned definite and distinct numeric values to "a", "b", and "c" and if "s" is a finite stream of those exact values, then
counts = histc(s, sort([a,b,c]));
probs = counts ./ length(s);
nzprobs = probs(probs>0);
entropy = sum(nzprobs .* log2(1./nzprobs));
Note: this code will work even if the assigned values are characters and s is a string.

rakesh ranjan on 17 Nov 2020
function Y=esingle(p)
if(p==0)
Y=0;
return;
else
Y=p*log2(1/p);
end
end