[R] manipulating large data and lmer
Katherine Jones
kajones at connect.carleton.ca
Wed Oct 31 16:47:37 CET 2007
I have switched to R having lost use of SPSS. Unfortunately all my
data was given to me by collaborators in SPSS files and the datasets
are too big to put into excel and manipulate (100,000 records).
I am managing to import my data into R from SPSS with foreign, with
no problems. I can do a hierarchical partitioning of variance
analysis on the data and do basic stuff like calculate means.
I'm now trying to do mixed models with lmer (lme4 package). I can do
the model, get the results (I realise it doesn't give p-values, not
quite got my head round this yet, but am aware of authors post on why
in R wiki; will need to read more about mixed models to fully
understand his answer) and do some basic residuals plots. e.g. I can
plot residuals versus fitted values.
My PROBLEM:-
-I can't plot residuals or fitted values against any of the
variables. I think this is because the model and dataset are of
different lengths due to lots of NA values. For my model, I specify
na.exclude. Ideally I'd like to remove all the NA data at the
read.table stage but I can't get it to do this. I also think I might
be using the wrong code (am use lme examples in the R book by Crawley).
-I wish to extract fitted values for certain factors in my model e.g.
sex. At the moment I can only get all of the fitted values, and not
subset them.
I think perhaps I have dived in rather deep into stats given my
sparse knowledge of R and of mixed models, but unfortunately I don't
have use of SPSS or SAS to play with, so have no choice.
Any help is much welcomed. I am slowly overcoming my fear of no menus
and starting to see the softwares potential.
Dr Katherine Jones
Department of Biology
Carleton University
More information about the R-help
mailing list