[R] User-defined functions in dplyr

Axel Urbiz axel.urbiz at gmail.com
Tue Nov 3 03:07:51 CET 2015


Nice example of the issue Bill. Thank you.

Is this a known issue? Plans to be fixed?

Thanks again,
Axel.


> On Nov 2, 2015, at 8:58 PM, William Dunlap <wdunlap at tibco.com> wrote:
> 
> dplyr::mutate does not collapse factor variables well.  They seem to get their levels from the levels
> computed for the first group and mutate does not check for them having different levels.
> 
> > data.frame(group=rep(c("A","B","C"),each=2), value=rep(c("X","Y","Z"),3:1)) %>% dplyr::group_by(group) %>% dplyr::mutate(fv=factor(value)) 
> Source: local data frame [6 x 3]
> Groups: group [3]
> 
>    group  value     fv
>   (fctr) (fctr) (fctr)
> 1      A      X      X
> 2      A      X      X
> 3      B      X      X
> 4      B      Y     NA
> 5      C      Y      X
> 6      C      Z     NA
> > levels(.Last.value$fv)
> [1] "X"
> 
> 
> 
> Bill Dunlap
> TIBCO Software
> wdunlap tibco.com <http://tibco.com/>
> On Mon, Nov 2, 2015 at 5:38 PM, Axel Urbiz <axel.urbiz at gmail.com <mailto:axel.urbiz at gmail.com>> wrote:
> Actually, the results are not the same. Looks like in the code below (see "using dplyr”), the function create_bins2 is not being applied separately to each "group_by" variable. That is surprising to me, or I'm misunderstanding dplyr.
> 
> ### Create some data
> 
> set.seed(4)
> df <- data.frame(pred = rnorm(100), models = gl(2, 50, 100, labels = c("model1", "model2")))
> 
> ### This is the code using plyr, which I'd like to change using dplyr
> 
> create_bins <- function(x, nBins) {
>   Breaks <- unique(quantile(x$pred, probs = seq(0, 1, 1/nBins)))
>   dfB <-  data.frame(pred = x$pred,
>                                 bin = cut(x$pred, breaks = Breaks, include.lowest = TRUE))
>   dfB
> }
> 
> nBins = 10
> res_plyr <- plyr::ddply(df, plyr::.(models), create_bins, nBins)
> head(res_plyr)
> 
> ### Attempt using dplyr
> 
> create_bins2 <- function (pred, nBins) {
>   Breaks <- unique(quantile(pred, probs = seq(0, 1, 1/nBins)))
>   bin <- cut(pred, breaks = Breaks, include.lowest = TRUE)
>   bin
> }
> 
> res_dplyr <- dplyr::mutate(dplyr::group_by(df, models),
>                                           bin=create_bins2(pred, nBins))
> 
> 
> identical(res_plyr, as.data.frame(res_dplyr))
> [1] FALSE
> #levels(res_dplyr$bin) == levels(res_plyr$bin)
> 
> Thanks,
> Axel.
> 
> 
> 
>> On Oct 30, 2015, at 12:19 PM, William Dunlap <wdunlap at tibco.com <mailto:wdunlap at tibco.com>> wrote:
>> 
>> dplyr::mutate is probably what you want instead of dplyr::summarize:
>> 
>> create_bins3 <- function (xpred, nBins) 
>> {
>>     Breaks <- unique(quantile(xpred, probs = seq(0, 1, 1/nBins)))
>>     bin <- cut(xpred, breaks = Breaks, include.lowest = TRUE)
>>     bin
>> }
>> dplyr::group_by(df, models) %>% dplyr::mutate(Bin=create_bins3(pred,nBins))
>> #Source: local data frame [100 x 3]
>> #Groups: models [2]
>> #
>> #         pred models               Bin
>> #        (dbl) (fctr)            (fctr)
>> #1   0.2167549 model1     (0.167,0.577]
>> #2  -0.5424926 model1   (-0.869,-0.481]
>> ...
>> 
>> 
>> Bill Dunlap
>> TIBCO Software
>> wdunlap tibco.com <http://tibco.com/>
>> On Fri, Oct 30, 2015 at 9:06 AM, William Dunlap <wdunlap at tibco.com <mailto:wdunlap at tibco.com>> wrote:
>> The error message is not very helpful and the stack trace is pretty inscrutable as well
>> > dplyr::group_by(df, models) %>% dplyr::summarize(create_bins)
>> Error: not a vector
>> > traceback()
>> 14: stop(list(message = "not a vector", call = NULL, cppstack = NULL))
>> 13: .Call("dplyr_summarise_impl", PACKAGE = "dplyr", df, dots)
>> 12: summarise_impl(.data, dots)
>> 11: summarise_.tbl_df(.data, .dots = lazyeval::lazy_dots(...))
>> 10: summarise_(.data, .dots = lazyeval::lazy_dots(...))
>> 9: dplyr::summarize(., create_bins)
>> 8: function_list[[k]](value)
>> 7: withVisible(function_list[[k]](value))
>> 6: freduce(value, `_function_list`)
>> 5: `_fseq`(`_lhs`)
>> 4: eval(expr, envir, enclos)
>> 3: eval(quote(`_fseq`(`_lhs`)), env, env)
>> 2: withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
>> 1: dplyr::group_by(df, models) %>% dplyr::summarize(create_bins)
>> 
>> 
>> It does not mean that your function, create_bins, does not return a vector --
>> the sum function gives the same result. help(summarize,package="dplyr")
>> says:
>>      ...: Name-value pairs of summary functions like ‘min()’, ‘mean()’,
>>           ‘max()’ etc.
>> It apparently means calls to summary functions, not summary functions
>> themselves.  The examples in the help file show the proper usage.
>> 
>> Use a call to your function and you will see it works better
>>    > dplyr::group_by(df, models) %>% dplyr::summarize(create_bins(pred,nBins))
>>    Error: $ operator is invalid for atomic vectors
>> The traceback again is not very useful, because the call information was
>> stripped by dplyr (by the call=NULL in the call to stop()):  
>>   > traceback()
>>   14: stop(list(message = "$ operator is invalid for atomic vectors", 
>>           call = NULL, cppstack = NULL))
>>   13: .Call("dplyr_summarise_impl", PACKAGE = "dplyr", df, dots)
>> However it is clear that the fault is in your function, which is expecting a
>> data.frame x with a column called pred but gets pred itself.  Change x to xpred
>> in the argument list and x$pred to xpred in the body of the function.
>> 
>> You will run into more problems because your function returns a vector
>> the length of its input but summarize expects a summary function - one
>> that returns a scalar for any size vector input.
>> 
>> Bill Dunlap
>> TIBCO Software
>> wdunlap tibco.com <http://tibco.com/>
>> 
>> On Fri, Oct 30, 2015 at 4:04 AM, Axel Urbiz <axel.urbiz at gmail.com <mailto:axel.urbiz at gmail.com>> wrote:
>> So in this case, "create_bins" returns a vector and I still get the same
>> error.
>> 
>> 
>> create_bins <- function(x, nBins)
>> {
>>   Breaks <- unique(quantile(x$pred, probs = seq(0, 1, 1/nBins)))
>>   bin <- cut(x$pred, breaks = Breaks, include.lowest = TRUE)
>>   bin
>> }
>> 
>> 
>> ### Using dplyr (fails)
>> nBins = 10
>> by_group <- dplyr::group_by(df, models)
>> res_dplyr <- dplyr::summarize(by_group, create_bins, nBins)
>> Error: not a vector
>> 
>> On Thu, Oct 29, 2015 at 8:28 PM, Jeff Newmiller <jdnewmil at dcn.davis.ca.us <mailto:jdnewmil at dcn.davis.ca.us>>
>> wrote:
>> 
>> > You are jumping the gun (your other email did get through) and you are
>> > posting using HTML (which does not come through on the list). Some time
>> > (re)reading the Posting Guide mentioned at the bottom of all emails on this
>> > list seems to be in order.
>> >
>> > The error is actually quite clear. You should return a vector from your
>> > function, not a data frame.
>> > ---------------------------------------------------------------------------
>> > Jeff Newmiller                        The     .....       .....  Go Live...
>> > DCN:<jdnewmil at dcn.davis.ca.us <mailto:jdnewmil at dcn.davis.ca.us>>        Basics: ##.#.       ##.#.  Live
>> > Go...
>> >                                       Live:   OO#.. Dead: OO#..  Playing
>> > Research Engineer (Solar/Batteries            O.O#.       #.O#.  with
>> > /Software/Embedded Controllers)               .OO#.       .OO#.  rocks...1k
>> > ---------------------------------------------------------------------------
>> > Sent from my phone. Please excuse my brevity.
>> >
>> > On October 29, 2015 4:55:19 PM MST, Axel Urbiz <axel.urbiz at gmail.com <mailto:axel.urbiz at gmail.com>>
>> > wrote:
>> > >Hello,
>> > >
>> > >Sorry, resending this question as the prior was not sent properly.
>> > >
>> > >I’m using the plyr package below to add a variable named "bin" to my
>> > >original data frame "df" with the user-defined function "create_bins".
>> > >I'd
>> > >like to get similar results using dplyr instead, but failing to do so.
>> > >
>> > >set.seed(4)
>> > >df <- data.frame(pred = rnorm(100), models = gl(2, 50, 100, labels =
>> > >c("model1", "model2")))
>> > >
>> > >
>> > >### Using plyr (works fine)
>> > >create_bins <- function(x, nBins)
>> > >{
>> > >  Breaks <- unique(quantile(x$pred, probs = seq(0, 1, 1/nBins)))
>> > >  dfB <-  data.frame(pred = x$pred,
>> > >                    bin = cut(x$pred, breaks = Breaks, include.lowest =
>> > >TRUE))
>> > >  dfB
>> > >}
>> > >
>> > >nBins = 10
>> > >res_plyr <- plyr::ddply(df, plyr::.(models), create_bins, nBins)
>> > >head(res_plyr)
>> > >
>> > >### Using dplyr (fails)
>> > >
>> > >by_group <- dplyr::group_by(df, models)
>> > >res_dplyr <- dplyr::summarize(by_group, create_bins, nBins)
>> > >Error: not a vector
>> > >
>> > >
>> > >Any help would be much appreciated.
>> > >
>> > >Best,
>> > >Axel.
>> > >
>> > >       [[alternative HTML version deleted]]
>> > >
>> > >______________________________________________
>> > >R-help at r-project.org <mailto:R-help at r-project.org> mailing list -- To UNSUBSCRIBE and more, see
>> > >https://stat.ethz.ch/mailman/listinfo/r-help <https://stat.ethz.ch/mailman/listinfo/r-help>
>> > >PLEASE do read the posting guide
>> > >http://www.R-project.org/posting-guide.html <http://www.r-project.org/posting-guide.html>
>> > >and provide commented, minimal, self-contained, reproducible code.
>> >
>> >
>> 
>>         [[alternative HTML version deleted]]
>> 
>> ______________________________________________
>> R-help at r-project.org <mailto:R-help at r-project.org> mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help <https://stat.ethz.ch/mailman/listinfo/r-help>
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html <http://www.r-project.org/posting-guide.html>
>> and provide commented, minimal, self-contained, reproducible code.
>> 
>> 
> 
> 


	[[alternative HTML version deleted]]



More information about the R-help mailing list