[R] R runtime performance and memory usage

[William Dunlap]

If a quick running time is important and your models involve only
numeric data with no missing values and you are willing to spend more
programming time setting things up, the lsfit() function may work
better for you.
Bill Dunlap
TIBCO Software
wdunlap tibco.com


On Mon, Nov 16, 2015 at 3:25 PM, Sasikumar Kandhasamy <ckmsasi at gmail.com> wrote:
> Thanks a lot Bill & Bert.
>
> Hi Bill,
>
> Sorry i was wrong on number of records, actually, i am using two dimensional
> data of 250K records each. And regarding CPU usage, it was the elapsed time.
> Infact, i have pined one core to run R.
>
> Thanks & Regards
> Sasi
>
> On Mon, Nov 16, 2015 at 2:04 PM, William Dunlap <wdunlap at tibco.com> wrote:
>>
>> You cannot do a linear regression with one column of data - there must
>> be at least one response column and one predictor.  By default, lm
>> throws in a constant term which gives you a second predictor.  If your
>> predictor is categorical, you get a new column for all but the first
>> unique value in it.
>>
>> lm() deals only with double precision data, at 8 bytes/number.  Thus
>> 250k numbers occupies 2 million bytes.  Your three columns (in the
>> non-categorical-predictor case)  take up 6 million bytes,
>>
>> lm()'s output contains several columns the size of the response
>> variable: residuals, effects, and fitted.values.  It also contains the
>> QR decomposition of the design matrix (the size of all the predictor
>> columns together).
>>
>> There are also some temporary variables generated in the course of the
>> computation.
>>
>> So your observed 40 MB memory usage seems reasonable.
>>
>> Use the object.size() function to see how big objects are and str() to
>> look at their structure.
>>
>> My laptop with  a 2.5 GHz Intel i7 processor takes a quarter second to
>> fit a simple linear model with one numeric predictor and a constant
>> term.  6 seconds sounds slow.  Is that cpu or elapsed time (use
>> system.time() to see)?
>>
>>
>>
>> Bill Dunlap
>> TIBCO Software
>> wdunlap tibco.com
>>
>>
>> On Mon, Nov 16, 2015 at 12:25 PM, Sasikumar Kandhasamy
>> <ckmsasi at gmail.com> wrote:
>> > Hi All,
>> >
>> > I have couple of clarifications on R run-time performance. I have
>> > R-3.2.2
>> > package compiled for MIPS64 and am running it on my linux machine with
>> > mips64 processor (core speed 1.5GHz) and observing the following
>> > behaviors,
>> >
>> > 1. Applying "linear regression model" (lm) on 1MB of data (contains 1
>> > column of 250K records) takes ~6 seconds to complete. Anyidea, is it an
>> > expected behavior or not? If not, can you please the suggestions or
>> > options
>> > to improve if we have any?
>> >
>> > 2. Also, the R process runtime virtual memory is increased by 40MB after
>> > applying the linear model on 1MB data. Is it also expected behavior? If
>> > it
>> > is expected, can you please share the insight of memory usage?
>> >
>> > Thanks in advance.
>> >
>> > Regards
>> > Sasi
>> >
>> >         [[alternative HTML version deleted]]
>> >
>> > ______________________________________________
>> > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> > https://stat.ethz.ch/mailman/listinfo/r-help
>> > PLEASE do read the posting guide
>> > http://www.R-project.org/posting-guide.html
>> > and provide commented, minimal, self-contained, reproducible code.
>
>