[R-pkg-devel] handling of byte-order-mark on r-devel-linux-x86_64-debian-clang machine

[Tomas Kalibera]

On 3/28/22 13:16, Ivan Krylov wrote:
> On Mon, 28 Mar 2022 09:54:57 +0200
> Tomas Kalibera <tomas.kalibera using gmail.com> wrote:
>
>> Could you please clarify which part you found somewhat confusing,
>> could that be improved?
> Perhaps "somewhat confusing" is an overstatement, sorry about that. All
> the information is already there in both ?file and ?readLines, it just
> requires a bit of thought to understand it.
>
>>> When reading from a text connection, the connections code, after
>>> re-encoding based on the ‘encoding’ argument, returns text that is
>>> assumed to be in native encoding; an encoding mark is only added by
>>> functions that read from the connection, so e.g.  ‘readLines’ can
>>> be instructed to mark the text as ‘"UTF-8"’ or ‘"latin1"’, but
>>> ‘readLines’ does no further conversion.  To allow reading text in
>>> ‘"UTF-8"’ on a system that cannot represent all such characters in
>>> native encoding (currently only Windows), a connection can be
>>> internally configured to return the read text in UTF-8 even though
>>> it is not the native encoding; currently ‘readLines’ and ‘scan’ use
>>> this feature when given a connection that is not yet open and, when
>>> using the feature, they unconditionally mark the text as ‘"UTF-8"’.
> The paragraph starts by telling the user that the text is decoded into
> the native encoding, then tells about marking the encoding (which is
> counter-productive when decoding arbitrarily-encoded text into native
> encoding) and only then presents the exception to the native encoding
> output rule (decoding into UTF-8). If I'm trying to read a
> CP1252-encoded file on a Windows 7 machine with CP1251 as the session
> encoding, I might get confused by the mention of encoding mark between
> the parts that are important to me.
>
> It could be an improvement to mention that exception closer to the
> first point of the paragraph and, perhaps, to split the "encoding mark"
> part from the "text connection decoding" part:
>
>>> Functions that read from the connection can add an encoding mark
>>> to the returned text. For example, ‘readLines’ can be instructed
>>> to mark the text as ‘"UTF-8"’ or ‘"latin1"’, but does no further
>>> conversion.
>>>
>>> When given a connection that is not yet open and has a non-default
>>> ‘encoding’ argument, ‘readLines’ and ‘scan’ internally configure the
>>> connection to read text in UTF-8. Otherwise, the text after decoding
>>> is assumed to be in native encoding.
> (Maybe this is omitting too much and should be expanded.)
>
> It could also be helpful to mention the fact that the encoding argument
> to readLines() can be ignored right in the description of that
> argument, inviting the user to read the Details section for more
> information.

Thanks for the suggestions, I've rewritten the paragraphs, biasing 
towards users who have UTF-8 as the native encoding as this is going to 
be the majority. These users should not have to worry much about the 
encoding marks anymore, nor about the internal UTF-8 mode of the 
connections code. But the level of detail I think needs to remain as 
long as these features are supported - the level of detail is based on 
numerous questions and bug reports.

Best
Tomas