# Calculating the entropy of a discrete memoryless source?

33 views (last 30 days)
Daniel_DG on 25 May 2015
Answered: rakesh ranjan on 17 Nov 2020
How could I calculate the entropy of a discrete memoryless source S={a,b,c} in unit of bits? What would be the matlab script for calculating this?
##### 0 CommentsShowHide -1 older comments

Sign in to comment.

### Answers (3)

Vinod Sudheesh on 27 May 2015
Hello Daniel,
You could write the script for the same as follows
1) Create a function, say "esingle" that computes the "entropy" of a symbol in the source alphabet. The definition of the function "esingle" could be as follows
function Y=esingle(p)
if(p==0)
Y=0;
return;
else
Y=p*log2(1/p);
end
end
2) Now invoke this function "esingle" using the syntax below to calculate the "entropy" of the source alphabet "a".
a=[0 1 1];
E=sum(arrayfun(@esingle,a/sum(a)));
Hope this helps!
Thanks Vinod
##### 1 CommentShowHide None
Daniel_DG on 27 May 2015
Thank you very much Vinod. It helped me a lot.

Sign in to comment.

Walter Roberson on 27 May 2015
A source S={a,b,c} is a source with three alphabetic symbols of no particular numeric value. If we know the generating equations for S then we analyze it analytically to determine the entropy. Otherwise the best we can do is estimate the entropy from a stream of the generated symbols. If we have assigned definite and distinct numeric values to "a", "b", and "c" and if "s" is a finite stream of those exact values, then
counts = histc(s, sort([a,b,c]));
probs = counts ./ length(s);
nzprobs = probs(probs>0);
entropy = sum(nzprobs .* log2(1./nzprobs));
Note: this code will work even if the assigned values are characters and s is a string.
##### 0 CommentsShowHide -1 older comments

Sign in to comment.

rakesh ranjan on 17 Nov 2020
function Y=esingle(p)
if(p==0)
Y=0;
return;
else
Y=p*log2(1/p);
end
end
##### 0 CommentsShowHide -1 older comments

Sign in to comment.

### Categories

Find more on Speech Recognition in Help Center and File Exchange

### Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by