[R] calculation of entropy in R???

David Reinke dreinke at dowlinginc.com
Tue Jul 8 23:31:35 CEST 2008

I've usually done my own calculations of entropy. Suppose p is a vector of
values (counts of outcomes, probabilities, etc.). Here's a simple function
to compute Shannon entropy:

shannon.entropy <- function(p)
	if (min(p) < 0 || sum(p) <= 0)
	p.norm <- p[p>0]/sum(p)

The p > 0 qualifier is put in to avoid taking log of zero.

It gets a bit more complicated if you're trying to find things like
conditional entropy and mutual information, e.g., H(X|Y), I(X;Y), but the
above is a basic building block for anything more complex. My approach for
getting H(X|Y), etc. is to get a contingency table, then calculate
conditional entropy and mutual information from the table. You might also
consult a text such as Cover & Thomas, Elements of Information Theory.


David Reinke

Senior Transportation Engineer/Economist
Dowling Associates, Inc.
180 Grand Avenue, Suite 250
Oakland, California 94612-3774
510.839.1742 x104 (voice)
510.839.0871 (fax)

-----Original Message-----
From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On
Behalf Of muhammad faisal
Sent: Tuesday, July 08, 2008 1:14 PM
To: R Help Cran
Subject: [R] calculation of entropy in R???

 i want to calculate shannon entropy which is 
        H1,H2,H3....upto H7?

if there is any function or any package in 
        which i can find this entropy directly. do you have any information

        please share this and i will be very thankful to you.
MUHAMMAD FAISAL  Department of Statistics and Decion Support system, 
University of Vienna,Austria      Res. # +4314796696104 
Cell # +436503605156
Office # +431427738662

	[[alternative HTML version deleted]]

R-help at r-project.org mailing list
PLEASE do read the posting guide
and provide commented, minimal, self-contained, reproducible code.

More information about the R-help mailing list