[R] anyone know why package "RandomForest" na.roughfix is so slow??
Liaw, Andy
andy_liaw at merck.com
Fri Jul 2 15:30:59 CEST 2010
I'll incorporate some of these ideas into the next release. Thanks!
Best,
Andy
-----Original Message-----
From: h.wickham at gmail.com [mailto:h.wickham at gmail.com] On Behalf Of Hadley Wickham
Sent: Thursday, July 01, 2010 8:08 PM
To: Mike Williamson
Cc: Liaw, Andy; r-help
Subject: Re: [R] anyone know why package "RandomForest" na.roughfix is so slow??
Here's another version that's a bit easier to read:
na.roughfix2 <- function (object, ...) {
res <- lapply(object, roughfix)
structure(res, class = "data.frame", row.names = seq_len(nrow(object)))
}
roughfix <- function(x) {
missing <- is.na(x)
if (!any(missing)) return(x)
if (is.numeric(x)) {
x[missing] <- median.default(x[!missing])
} else if (is.factor(x)) {
freq <- table(x)
x[missing] <- names(freq)[which.max(freq)]
} else {
stop("na.roughfix only works for numeric or factor")
}
x
}
I'm cheating a bit because as.data.frame is so slow.
Hadley
On Thu, Jul 1, 2010 at 6:44 PM, Mike Williamson <this.is.mvw at gmail.com> wrote:
> Jim, Andy,
>
> Thanks for your suggestions!
>
> I found some time today to futz around with it, and I found a "home
> made" script to fill in NA values to be much quicker. For those who are
> interested, instead of using:
>
> dataSet <- na.roughfix(dataSet)
>
>
>
> I used:
>
> origCols <- names(dataSet)
> ## Fix numeric values...
> dataSet <- as.data.frame(lapply(dataSet, FUN=function(x)
> {
> if(!is.numeric(x)) { x } else {
> ifelse(is.na(x), median(x, na.rm=TRUE), x) } }
> ),
> row.names=row.names(dataSet) )
> ## Fix factors...
> dataSet <- as.data.frame(lapply(dataSet, FUN=function(x)
> {
> if(!is.factor(x)) { x } else {
> levels(x)[ifelse(!is.na
> (x),x,table(max(table(x)))
> ) ] } } ),
> row.names=row.names(dataSet) )
> names(dataSet) <- origCols
>
>
>
> In one case study that I ran, the na.roughfix() algo took 296 seconds
> whereas the homemade one above took 16 seconds.
>
> Regards,
> Mike
>
>
>
> "Telescopes and bathyscaphes and sonar probes of Scottish lakes,
> Tacoma Narrows bridge collapse explained with abstract phase-space maps,
> Some x-ray slides, a music score, Minard's Napoleanic war:
> The most exciting frontier is charting what's already here."
> -- xkcd
>
> --
> Help protect Wikipedia. Donate now:
> http://wikimediafoundation.org/wiki/Support_Wikipedia/en
>
>
> On Thu, Jul 1, 2010 at 10:05 AM, Liaw, Andy <andy_liaw at merck.com> wrote:
>
>> You need to isolate the problem further, or give more detail about your
>> data. This is what I get:
>>
>> R> nr <- 2134
>> R> nc <- 14037
>> R> x <- matrix(runif(nr*nc), nr, nc)
>> R> n.na <- round(nr*nc/10)
>> R> x[sample(nr*nc, n.na)] <- NA
>> R> system.time(x.fixed <- na.roughfix(x))
>> user system elapsed
>> 8.44 0.39 8.85
>> R 2.11.1, randomForest 4.5-35, Windows XP (32-bit), Thinkpad T61 with 2GB
>> ram.
>>
>> Andy
>>
>> ------------------------------
>> *From:* Mike Williamson [mailto:this.is.mvw at gmail.com]
>> *Sent:* Thursday, July 01, 2010 12:48 PM
>> *To:* Liaw, Andy
>> *Cc:* r-help
>> *Subject:* Re: [R] anyone know why package "RandomForest" na.roughfix is
>> so slow??
>>
>> Andy,
>>
>> You're right, I didn't supply any code, because my call was very simple
>> and it was the call itself at question. However, here is the associated
>> code I am using:
>>
>>
>> naFixTime <- system.time( {
>> if (fltrResponse) { ## TRUE: there are no NA's in the
>> response... cleared via earlier steps
>> message(paste(iAm,": Missing values will now be
>> imputed...\n", sep=""))
>> try( dataSet <- rfImpute(dataSet[,!is.element(names(dataSet),
>> response)],
>> dataSet[,response]) )
>> } else { ## In this case, there is no "response" column in the
>> data set
>> message(paste(iAm,": Missing values will now be filled in
>> with median",
>> " values or most frequent levels", sep=""))
>> try( dataSet <- na.roughfix(dataSet) )
>> }
>> } )
>>
>>
>>
>> As you can see, the "na.roughfix" call is made as simply as possible:
>> I supply the entire dataSet (only parameters, no responses). I am not doing
>> the prediction here (that is done later, and the prediction itself is not
>> taking very long).
>> Here are some calculation times that I experienced:
>>
>> # rows # cols time to run na.roughfix
>> ======= ======= ====================
>> 2046 2833 ~ 2 minutes
>> 2066 5626 ~ 6 minutes
>> 2134 14037 ~ 30 minutes
>>
>> These numbers are on a Windows server using the 64-bit version of 'R'.
>>
>> Regards,
>> Mike
>>
>>
>> "Telescopes and bathyscaphes and sonar probes of Scottish lakes,
>> Tacoma Narrows bridge collapse explained with abstract phase-space maps,
>> Some x-ray slides, a music score, Minard's Napoleanic war:
>> The most exciting frontier is charting what's already here."
>> -- xkcd
>>
>> --
>> Help protect Wikipedia. Donate now:
>> http://wikimediafoundation.org/wiki/Support_Wikipedia/en
>>
>>
>> On Thu, Jul 1, 2010 at 8:58 AM, Liaw, Andy <andy_liaw at merck.com> wrote:
>>
>>> You have not shown any code on exactly how you use na.roughfix(), so I
>>> can only guess.
>>>
>>> If you are doing something like:
>>>
>>> randomForest(y ~ ., mybigdata, na.action=na.roughfix, ...)
>>>
>>> I would not be surprised that it's taking very long on large datasets.
>>> Most likely it's caused by the formula interface, not na.roughfix()
>>> itself.
>>>
>>> If that is your case, try doing the imputation beforehand and run
>>> randomForest() afterward; e.g.,
>>>
>>> myroughfixed <- na.roughfix(mybigdata)
>>> randomForest(myroughfixed[list.of.predictor.columns],
>>> myroughfixed[[myresponse]],...)
>>>
>>> HTH,
>>> Andy
>>>
>>> -----Original Message-----
>>> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org]
>>> On Behalf Of Mike Williamson
>>> Sent: Wednesday, June 30, 2010 7:53 PM
>>> To: r-help
>>> Subject: [R] anyone know why package "RandomForest" na.roughfix is so
>>> slow??
>>>
>>> Hi all,
>>>
>>> I am using the package "random forest" for random forest
>>> predictions. I
>>> like the package. However, I have fairly large data sets, and it can
>>> often
>>> take *hours* just to go through the "na.roughfix" call, which simply
>>> goes
>>> through and cleans up any NA values to either the median (numerical
>>> data) or
>>> the most frequent occurrence (factors).
>>> I am going to start doing some comparisons between na.roughfix() and
>>> some apply() functions which, it seems, are able to do the same job more
>>> quickly. But I hesitate to duplicate a function that is already in the
>>> package, since I presume the na.roughfix should be as quick as possible
>>> and
>>> it should also be well "tailored" to the requirements of random forest.
>>>
>>> Has anyone else seen that this is really slow? (I haven't noticed
>>> rfImpute to be nearly as slow, but I cannot say for sure: my "predict"
>>> data
>>> sets are MUCH larger than my model data sets, so cleaning the prediction
>>> data set simply takes much longer.)
>>> If so, any ideas how to speed this up?
>>>
>>> Thanks!
>>> Mike
>>>
>>>
>>>
>>> "Telescopes and bathyscaphes and sonar probes of Scottish lakes,
>>> Tacoma Narrows bridge collapse explained with abstract phase-space maps,
>>> Some x-ray slides, a music score, Minard's Napoleanic war:
>>> The most exciting frontier is charting what's already here."
>>> -- xkcd
>>>
>>> --
>>> Help protect Wikipedia. Donate now:
>>> http://wikimediafoundation.org/wiki/Support_Wikipedia/en
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> R-help at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>> Notice: This e-mail message, together with any attachments, contains
>>> information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
>>> New Jersey, USA 08889), and/or its affiliates Direct contact information
>>> for affiliates is available at
>>> http://www.merck.com/contact/contacts.html) that may be confidential,
>>> proprietary copyrighted and/or legally privileged. It is intended solely
>>> for the use of the individual or entity named on this message. If you are
>>> not the intended recipient, and have received this message in error,
>>> please notify us immediately by reply e-mail and then delete it from
>>> your system.
>>>
>>>
>> Notice: This e-mail message, together with any attach...{{dropped:15}}
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
--
Assistant Professor / Dobelman Family Junior Chair
Department of Statistics / Rice University
http://had.co.nz/
Notice: This e-mail message, together with any attachme...{{dropped:11}}
More information about the R-help
mailing list