[R] Limit

[Jeff Newmiller]

There is always an implied "and do computations on it before writing the processed data out" when reading chunks of a file.

And you would almost certainly not be getting that error if you were not out of memory.  A good rule of thumb is that you need 4 times as much free memory to process data than you need to read it in.

On November 8, 2024 6:08:16 PM PST, Val <valkremk using gmail.com> wrote:
>Hi Jeff,
>
>Memory was not an issue. The system only used 75% of the memory
>allocated for the job.
>
> I am trying to understand what  "r read large file in chunks" is doing.
>
>On Fri, Nov 8, 2024 at 7:50 PM Jeff Newmiller <jdnewmil using dcn.davis.ca.us> wrote:
>>
>> Then you don't have enough memory to process the whole thing at once. Not unlike stuffing your mouth with cookies and not being able to chew for lack of space to move the food around in your mouth.
>>
>> Now, can you answer my question?
>>
>> On November 8, 2024 5:38:37 PM PST, Val <valkremk using gmail.com> wrote:
>> >The data was read. The problem is with processing.
>> >
>> >On Fri, Nov 8, 2024 at 7:30 PM Bert Gunter <bgunter.4567 using gmail.com> wrote:
>> >>
>> >> Is the problem reading the file in or processing it after it has been read in?
>> >>
>> >> Bert
>> >>
>> >> On Fri, Nov 8, 2024 at 5:13 PM Jeff Newmiller via R-help <r-help using r-project.org> wrote:
>> >>>
>> >>> Can you tell us what is wrong with the "chunked" package which comes up when you Google "r read large file in chunks"?
>> >>>
>> >>> On November 8, 2024 4:58:18 PM PST, Val <valkremk using gmail.com> wrote:
>> >>> >Hi All,
>> >>> >
>> >>> >I am reading data file ( > 1B rows) and do some date formatting like
>> >>> >      dat=fread(mydatafile)
>> >>> >     dat$date1 <- as.Date(ymd(dat$date1))
>> >>> >
>> >>> >However, I am getting an error message saying that
>> >>> >    Error: cons memory exhausted (limit reached?)
>> >>> >
>> >>> >The  script was working  when the number rows were  around 650M.
>> >>> >
>> >>> >Is there another way to  handle  a big data set in R?
>> >>> >
>> >>> >
>> >>> >Thank you.
>> >>> >
>> >>> >______________________________________________
>> >>> >R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> >>> >https://stat.ethz.ch/mailman/listinfo/r-help
>> >>> >PLEASE do read the posting guide https://www.R-project.org/posting-guide.html
>> >>> >and provide commented, minimal, self-contained, reproducible code.
>> >>>
>> >>> --
>> >>> Sent from my phone. Please excuse my brevity.
>> >>>
>> >>> ______________________________________________
>> >>> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> >>> https://stat.ethz.ch/mailman/listinfo/r-help
>> >>> PLEASE do read the posting guide https://www.R-project.org/posting-guide.html
>> >>> and provide commented, minimal, self-contained, reproducible code.
>>
>> --
>> Sent from my phone. Please excuse my brevity.

-- 
Sent from my phone. Please excuse my brevity.