[R] logistic discrimination: which chance performance??
Bruno L. Giordano
bruno.giordano at music.mcgill.ca
Fri Aug 11 00:18:22 CEST 2006
Hello,
I am using logistic discriminant analysis to check whether a known
classification Yobs can be predicted by few continuous variables X.
What I do is to predict class probabilities with multinom() in nnet(),
obtaining a predicted classification Ypred and then compute the percentage
P(obs) of objects classified the same in Yobs and Ypred.
My problem now is to figure out whether P(obs) is significantly higher than
chance.
I opted for a crude permutation approach: compute P(perm) over 10000 random
permutations of Yobs (i.e., refit the multinom() model 10000 times randomly
permuting Yobs) and consider P(obs) as significantly higher than chance if
higher than the 95th percentile of the P(perm) distribution.
Now, the problem is that the mode of P(perm) is always really close to
P(obs), e.g., if P(obs)=1 (perfect discrimination) also the most likely
P(perm) value is 1!!!
I figured out that this is due to the fact that, with my data, randomly
permuted classifications are highly likely to strongly agree with the
observed classification Yobs, but, probably since my machine learning
background is almost 0, I am kind of lost about how to proceed at this
point.
I would greatly appreciate a comment on this.
Thanks
Bruno
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Bruno L. Giordano, Ph.D.
CIRMMT
Schulich School of Music, McGill University
555 Sherbrooke Street West
Montréal, QC H3A 1E3
Canada
http://www.music.mcgill.ca/~bruno/
More information about the R-help
mailing list