[R] Error because of large dimension

[li li]

Thanks all for the reply. I think I need to think of other ways to approach
the problem.
    Hanna

2016-01-24 17:45 GMT-05:00 William Dunlap <wdunlap at tibco.com>:

> > 28 PiB.  Storing such a large matrix even on file is not possible.
>
> The ads for Amazon Red Shift say it is possible.  E.g.,
>   Amazon Redshift is a fast, fully managed, petabyte-scale data
>   warehouse that makes it simple and cost-effective to analyze
>   all your data using your existing business intelligence tools.
>   Start small for $0.25 per hour with no commitments and scale
>   to petabytes for $1,000 per terabyte per year, less than a tenth
>   the cost of traditional solutions. Customers typically see 3x
>   compression, reducing their costs to $333 per uncompressed
>   terabyte per year.
>
> Cost may be an issue:
>
> 28 petabytes * 1024 petabytes/terabyte * $333 terabyte/year ~= $9.5
> million/year
> or $26 thousand/day.
>
>
> Bill Dunlap
> TIBCO Software
> wdunlap tibco.com
>
> On Sun, Jan 24, 2016 at 1:29 PM, Henrik Bengtsson <
> henrik.bengtsson at gmail.com> wrote:
>
>> FYI, the matrix you tried to allocate would hold
>> (3195*1290*495*35*35*35*15) * 3 = 3.936248e+15 values.  Each value
>> would occupy 8 bytes of memory (for the double data type).  In other
>> words, in order to keep this data matrix in memory you would require a
>> computer with at least 3.148998e+16 bytes of RAM, i.e. 29327331 GiB =
>> 28640 TiB = 28 PiB.  Storing such a large matrix even on file is not
>> possible.
>>
>> In other words, you need to figure out how to approach your original
>> problem in a different way.
>>
>> /Henrik
>>
>> On Sun, Jan 24, 2016 at 8:46 AM, li li <hannah.hlx at gmail.com> wrote:
>> > Hi all,
>> >   I am doing some calculation with very large dimension. I need to
>> create a
>> > matrix
>> > with three columns and a very large number of rows
>> > (3195*1290*495*35*35*35*15=1.312083e+15) i
>> > n order to allocate calculation result from a for loop.
>> > R does not allow me to create such a matrix because of the large
>> dimension
>> > (see below). Is there a way to go around this?
>> >   Thanks very much!!
>> >      Hanna
>> >
>> >
>> >> matrix(0, 3195*1290*495*35*35*35*15, 3)
>> > Error in matrix(0, 3195 * 1290 * 495 * 35 * 35 * 35 * 15, 3) :
>> >   invalid 'nrow' value (too large or NA)
>> > In addition: Warning message:
>> > In matrix(0, 3195 * 1290 * 495 * 35 * 35 * 35 * 15, 3) :
>> >   NAs introduced by coercion
>> >>
>> >
>> >         [[alternative HTML version deleted]]
>> >
>> > ______________________________________________
>> > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> > https://stat.ethz.ch/mailman/listinfo/r-help
>> > PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> > and provide commented, minimal, self-contained, reproducible code.
>>
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>

	[[alternative HTML version deleted]]