[R] How to save very large matrix?

[Prof Brian Ripley]

On 29/10/2013 20:42, Rui Barradas wrote:
> Hello,
>
> You can use the argument to write.csv or write.table  append = TRUE to
> write the matrix in chunks. Something like the following.

That was going to be my suggestion. But the reason long vectors have not 
been implemented is that is rather implausible to be useful.   A text 
file with the values of such a numeric matrix is likely to be 100GB. 
What are you going to do with such a file?  For transfer to another 
program I would seriously consider a binary format (e.g. use writeBin), 
as it is the conversion to and from text that is time consuming.

Some experiments suggest that it would take hours to write and at least 
an hour to read such a file[*], on a very fast machine with a 
start-of-the-art SSD.

[*] a file with reasonable-precision real numbers, not zeroes.

>
>
>
> bigwrite <- function(x, file, rows = 1000L, ...){
>      passes <- NROW(x) %/% rows
>      remaining <- NROW(x) %% rows
>      k <- 1L
>      write.table(x[k:rows, ], file, row.names = FALSE, ...)
>      k <- k + rows
>      for(i in seq_len(passes)[-1]){
>          write.table(x[k:(rows*i), ], file, append = TRUE, row.names =
> FALSE, col.names = FALSE, ...)
>          k <- k + rows
>      }
>      if(remaining > 0)
>          write.table(x[k:NROW(x), ], file, append = TRUE, row.names =
> FALSE, col.names = FALSE, ...)
> }
>
> f <- "temp"
> m <- matrix(0, 50012, 10)
>
> bigwrite(m, f, sep = ",")  # Use 'sep' to get a csv file
>
>
>
> Hope this helps,
>
> Rui Barradas
>
>
> Em 29-10-2013 19:27, Petar Milin escreveu:
>> Hello!
>> I have a very large matrix of results: 50000x100000. I saved it as
>> RDS, but I would also need to save it as txt or csv. Is there a way to
>> do it? Now, with write.table I am receiving an error:
>> Error in .External2(C_writetable, x, file, nrow(x), p, rnames, sep,
>> eol,  :
>>    long vectors not supported yet: io.c:1116
>>
>> Please, help! Many thanks!
>>
>> PM


-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595