[Rd] duplicated factor labels.
Paul Johnson
pauljohn32 at gmail.com
Fri Jun 23 18:17:23 CEST 2017
On Fri, Jun 23, 2017 at 7:20 AM, Uwe Ligges
<ligges at statistik.tu-dortmund.de> wrote:
>
>
> On 23.06.2017 11:51, peter dalgaard wrote:
>>
>> Hmm, the danger in this is that duplicated factor levels _used_ to be
>> allowed (i.e. multiple codes with the same level). Disallowing it is what
>> broke read.spss() on some files, because SPSS's concept of value labels is
>> not 1-to-1 with factors.
>>
>> Reallowing it with different semantics could be premature. I mean, if we
>> hadn't had the "forbidden" step, read.spss() could have changed behaviour
>> unnoticed. So what if there is code relying on duplicate factor levels,
>> which hasn't been run for some time?
>
>
> Indeed.
>
> The read.spss code now allows for two things, one is to do what Martin
> implemented, the other one is to keep the labels seperated and rename them
> to be unique (the latter is the default now, explanation follows below).
>
> Quite often we found something like the following example in SPSS files,
> translated into R speak:
>
> factor(c(1,3,2,4,5,3,1), levels=1:5, labels=c("Strongly disagree",
> "Disagree", "Neither agree nor disagree", "Agree", "Agree"))
>
> where the last is a simple copy and paste error and should be "Strongly
> agree".
>
> I had the chance to look at > 1300 SPSS files our consulting center
> collected during the last 20 year, and in several hundred cases we found
> such a problem that was copy & paste error and simply wrong.
> Only in < 5 cases condensing several levels into one was appropriate, hence
> we decided to keep duplicated levels by changing the names as the default.
>
> Based on this experience I'd propose no to touch factor but rather add a
> function that easily allows for this reduction, if we do not have that
> already.
>
> Best,
> Uwe
>
>
If the factor function stays the way it was, I have a suggestion Uwe's
suggest R should add a function to facilitate reduction of labels.
There is a function named "mapvalues" in H Wickham's plyr package and
it works well to combine levels. It is a generally useful recoding
function, works with integers and characters as well. It also seems to
work with numeric variables. This is pure R, it does not carry along
with it any external dependencies. No need for Rcpp, %>% or any thing
else.
Of the things we have tried with the average users who come and go,
mapvalues is the most understandable/successful. It is more convenient
than levels()<- because Users need not name all existing levels.
I suggest you consider putting that function in R base.
I'm pasting in the code to save you the trouble of looking it up. I
thought recursion to re-code factors was clever. I don't entirely
understand how it can work on double precision floats, it is relying
on match for that.
#' Replace specified values with new values, in a vector or factor.
#'
#' Item in \code{x} that match items \code{from} will be replaced by
#' items in \code{to}, matched by position. For example, items in \code{x} that
#' match the first element in \code{from} will be replaced by the first
#' element of \code{to}.
#'
#' If \code{x} is a factor, the matching levels of the factor will be
#' replaced with the new values.
#'
#' The related \code{revalue} function works only on character vectors
#' and factors, but this function works on vectors of any type and factors.
#'
#' @param x the factor or vector to modify
#' @param from a vector of the items to replace
#' @param to a vector of replacement values
#' @param warn_missing print a message if any of the old values are
#' not actually present in \code{x}
#'
#' @seealso \code{\link{revalue}} to do the same thing but with a single
#' named vector instead of two separate vectors.
#' @export
#' @examples
#' x <- c("a", "b", "c")
#' mapvalues(x, c("a", "c"), c("A", "C"))
#'
#' # Works on factors
#' y <- factor(c("a", "b", "c", "a"))
#' mapvalues(y, c("a", "c"), c("A", "C"))
#'
#' # Works on numeric vectors
#' z <- c(1, 4, 5, 9)
#' mapvalues(z, from = c(1, 5, 9), to = c(10, 50, 90))
mapvalues <- function(x, from, to, warn_missing = TRUE) {
if (length(from) != length(to)) {
stop("`from` and `to` vectors are not the same length.")
}
if (!is.atomic(x)) {
stop("`x` must be an atomic vector.")
}
if (is.factor(x)) {
# If x is a factor, call self but operate on the levels
levels(x) <- mapvalues(levels(x), from, to, warn_missing)
return(x)
}
mapidx <- match(x, from)
mapidxNA <- is.na(mapidx)
# index of items in `from` that were found in `x`
from_found <- sort(unique(mapidx))
if (warn_missing && length(from_found) != length(from)) {
message("The following `from` values were not present in `x`: ",
paste(from[!(1:length(from) %in% from_found) ], collapse = ", "))
}
x[!mapidxNA] <- to[mapidx[!mapidxNA]]
x
}
In the rockchalk package, I wrote a function called combineLevels that
is careful with ordinal variables and only allows adjacent values to
be combined. I'm not suggesting you go that far with this simple
piece.
>
>
>
>
>>
>> -pd
>>
>>> On 23 Jun 2017, at 10:42 , Martin Maechler <maechler at stat.math.ethz.ch>
>>> wrote:
>>>
>>>>>>>> Martin Maechler <maechler at stat.math.ethz.ch>
>>>>>>>> on Thu, 22 Jun 2017 11:43:59 +0200 writes:
>>>
>>>
>>>>>>>> Paul Johnson <pauljohn32 at gmail.com>
>>>>>>>> on Fri, 16 Jun 2017 11:02:34 -0500 writes:
>>>
>>>
>>>>> On Fri, Jun 16, 2017 at 2:35 AM, Joris Meys <jorismeys at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> To extwnd on Martin 's explanation :
>>>>>>
>>>>>> In factor(), levels are the unique input values and labels the unique
>>>>>> output
>>>>>> values. So the function levels() actually displays the labels.
>>>>>>
>>>
>>>>> Dear Joris
>>>
>>>
>>>>> I think we agree. Currently, factor insists both levels and labels be
>>>>> unique.
>>>
>>>
>>>>> I wish that it would not accept nonunique labels. I also understand it
>>>>> is impractical to change this now in base R.
>>>
>>>
>>>>> I don't think I succeeded in explaining why this would be nicer.
>>>>> Here's another example. Fairly often, we see input data like
>>>
>>>
>>>>> x <- c("Male", "Man", "male", "Man", "Female")
>>>
>>>
>>>>> The first four represent the same value. I'd like to go in one step
>>>>> to a new factor variable with enumerated types "Male" and "Female".
>>>>> This fails
>>>
>>>
>>>>> xf <- factor(x, levels = c("Male", "Man", "male", "Female"),
>>>>> labels = c("Male", "Male", "Male", "Female"))
>>>
>>>
>>>>> Instead, we need 2 steps.
>>>
>>>
>>>>> xf <- factor(x, levels = c("Male", "Man", "male", "Female"))
>>>>> levels(xf) <- c("Male", "Male", "Male", "Female")
>>>
>>>
>>>>> I think it is quirky that `levels<-.factor` allows the duplicated
>>>>> labels, whereas factor does not.
>>>
>>>
>>>>> I wrote a function rockchalk::combineLevels to simplify combining
>>>>> levels, but most of the students here like plyr::mapvalues to do it.
>>>>> The use of levels() can be tricky because one must enumerate all
>>>>> values, not just the ones being changed.
>>>
>>>
>>>>> But I do understand Martin's point. Its been this way 25 years, it
>>>>> won't change. :).
>>>
>>>
>>>> Well.. the above is a bit out of context.
>>>
>>>
>>>> Your first example really did not make a point to me (and Joris)
>>>> and I showed that you could use even two different simple factor() calls
>>>> to
>>>> produce what you wanted
>>>> yc <- factor(c("1",NA,NA,"4","4","4"))
>>>> yn <- factor(c( 1, NA,NA, 4, 4, 4))
>>>
>>>
>>>> Your new example is indeed much more convincing !
>>>
>>>
>>>> (Note though that the two steps that are needed can be written
>>>> more shortly
>>>
>>>
>>>> The "been this way 25 years" is one a reason to be very
>>>> cautious(*) with changes, but not a reason for no changes!
>>>
>>>
>>>> (*) Indeed as some of you have noted we really should not "break
>>>> behavior".
>>>> This means to me we cannot accept a change there which gives
>>>> an error or a different result in cases the old behavior gave a valid
>>>> factor.
>>>
>>>
>>>> I'm looking at a possible change currently
>>>> [not promising that a change will happen ...]
>>>
>>>
>>> In the end, I've liked the change (after 2-3 iterations), and
>>> now been brave to commit to R-devel (svn 72845).
>>>
>>> With the change, I had to disable one of our own regression
>>> checks (tests/reg-tests-1b.R, line 726):
>>>
>>> The following is now (in R-devel -> R 3.5.0) valid:
>>>
>>>> factor(1:2, labels = c("A","A"))
>>>
>>> [1] A A
>>> Levels: A
>>>>
>>>>
>>>
>>> I wonder how many CRAN package checks will "break" from
>>> this (my guess is in the order of a dozen), but I hope
>>> that these breakages will be benign, e.g., similar to the above
>>> case where before an error was expected via tools :: assertError(.)
>>>
>>> Martin
>>>
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>>
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
--
Paul E. Johnson http://pj.freefaculty.org
Director, Center for Research Methods and Data Analysis http://crmda.ku.edu
To write to me directly, please address me at pauljohn at ku.edu.
More information about the R-devel
mailing list