[R] Failed to convert data to numeric

[@vi@e@gross m@iii@g oii gm@ii@com]

The second solution Ivan offers looks good, and a bit more general than his first that simply removes one non-visible character.

It begs the question of why the data has that anomaly at all. Did the data come from a text-processing environment where it was going to wrap there and was protected?

As Ivan points out, there is a question of what format you expect numbers in and what "as.numeric"  should do when it does not see an integer or floating point number. 

If you test it, you can see that as.numeric ignores leading and/or trailing blanks and tabs and even newlines sometimes and some other irrelevant ASCII characters. In that spirit, the UNICODE character being mentioned should be one that any UNICODE-aware version of as.numeric should ignore.

But UNICODE supports a much wider vision of numeric so that there are numeric-equivalent symbols in other languages and groupings and even something like the symbols for numerals in light or dark circles count as numbers. Those can likely safely be excluded in this context but perhaps not in a more general function.

But I note as.numeric seems to handle scientific notation as in:

as.numeric("1.23e8")
[1] 1.23e+08

So a single instance of the letters "e" and "E" must be supported if your numbers in string form may contain them. Further, the E cannot be the first or last letter. It cannot have adjacent whitespace. Still, if you are OK with getting an NA in such situations, it should be OK.

It gets worse. Hexadecimal is supported:

> as.numeric("0X12")
[1] 18

You now need to support the letters x and X. But only if preceded by a zero! 

It gets still worse as any characters from [0-9A-F] are supported:

> as.numeric("0xAE")
[1] 174

There may be other scenarios it handles. The filter applied might remove valid numbers so you may want to carefully document it if your program only handles a restricted set.

A possible idea might be to make two passes and only  evaluate any resulting NA from as.numeric() by doing a substitution like Ivan suggests to try to fix any broken ones. But note it may fix too much as "1.2 e 5" might become "1.2e5" as spaces are removed.

-----Original Message-----
From: R-help <r-help-bounces using r-project.org> On Behalf Of Ivan Krylov via R-help
Sent: Monday, March 3, 2025 3:09 AM
To: Christofer Bogaso <bogaso.christofer using gmail.com>
Cc: r-help <r-help using r-project.org>
Subject: Re: [R] Failed to convert data to numeric

В Mon, 3 Mar 2025 13:21:31 +0530
Christofer Bogaso <bogaso.christofer using gmail.com> пишет:

> Is there any way to remove all possible "Unicode character" that may
> be present in the array at once?

Define a range of characters you consider acceptable, and you'll be
able to use regular expressions to remove everything else. For example,
the following expression should remove everything except ASCII digits,
dots, and hyphen-minus:

gsub('[^0-9.-]+', '', dat2)

There is a brief introduction to regular expressions in ?regex and
various online resources such as <https://regex101.com/>.

-- 
Best regards,
Ivan

______________________________________________
R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide https://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.