[R] using mclapply (multi core apply) to do matrix multiplication

[Rainer M Krug]

On 07/02/12 12:02, Alaios wrote:
> Thank you very much for your point...
> I hope I can find some easy to follow instructions as I do not have root
> permission

Me neither on pur cluster - but that won't stop you from compiling and 
installing R in your home directory. By doing this, you have even more 
control.

Cheers and good luck,

Rainer

for the many cores system and our system administrator want
> to have easy instructions to follow.
> Thanks a gain.
>
> ------------------------------------------------------------------------
> *From:* Rainer M Krug <r.m.krug at gmail.com>
> *To:* Alaios <alaios at yahoo.com>
> *Cc:* Ernest Adrogué <nfdisco at gmail.com>; "r-help at r-project.org"
> <r-help at r-project.org>
> *Sent:* Tuesday, February 7, 2012 11:44 AM
> *Subject:* Re: [R] using mclapply (multi core apply) to do matrix
> multiplication
>
> On 07/02/12 11:31, Alaios wrote:
>  > I would like to thank you Ernest for your answer. I guess that this
>  > is gonna be faster as right now R only sees one core. In my work
>  > there is a system with 64 cores and you can see only one working. If
>  > I understand it right a [m,n][n,k] matrix multiplication can be split
>  > into rows (from first matrice) and columns (from the second matrice)
>  > and then combine all the local results of each cpu together.
>
> You definitaly can go this way, but I would STRONGLY recommend to search
> for "parallel BLAS", check in the R-admin manual the section "Linear
> Algebra" which deals with BLAS et al, and e.g.
> http://www.r-bloggers.com/compiling-64-bit-r-2-10-1-with-mkl-in-linux/
>
> My guess is that a paralelization on the C level in the BLAS et al.
> library will be MUCH faster then a paralelization on R level.
>
> Also, there is a R-sig-hpc mailing list for these kind of questions.
>
> Cheers,
>
> Rainer
>  >
>  > Would that be too weird for mclapply to handle?
>  >
>  > B.R Alex
>  >
>  >
>  >
>  > ________________________________ From: Ernest
>  > Adrogué<nfdisco at gmail.com <mailto:nfdisco at gmail.com>> To:
> r-help at r-project.org <mailto:r-help at r-project.org> Sent: Tuesday,
>  > February 7, 2012 11:02 AM Subject: Re: [R] using mclapply (multi core
>  > apply) to do matrix multiplication
>  >
>  > 7-02-2012, 00:29 (-0800); Alaios escriu:
>  >> Dear all, I am trying to multiply three different matrices and
>  >> each matrice is of size 16384,16384 the normal %*% multiplciation
>  >> operator has not finished one day now. As I am running a system
>  >> with many cores (and it seems that R is using only one of those) I
>  >> would like to write fast a brief function that converts the typical
>  >> for loops of a matrix multiplication to a set of lapply sets
>  >> (mclapply uses the lapply syntax but it applies the work to many
>  >> cores).
>  >>
>  >> If my thinking is correct , in the sense that this will speed up
>  >> things a lot, I want you to help me covert the first matrix in
>  >> rows the second in columns and convert those in a format that
>  >> lapply would like to work with.
>  >
>  > If I understand correctly, R uses a specialized library called BLAS
>  > to do matrix multiplications. I doubt re-implementing the matrix
>  > multiplication code at R-level would be any faster. What you can try
>  > is replace BLAS with a multicore version of BLAS although it's not
>  > easy if you have to compile it yourself.
>  >
>  > Also, you may try to re-think the problem you're trying to solve.
>  > Maybe there's a different approach that is less
>  > computation-intensive.
>  >
>  >
>  >
>  >
>  > ______________________________________________ R-help at r-project.org
> <mailto:R-help at r-project.org>
>  > mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do
>  > read the posting guide http://www.R-project.org/posting-guide.html
>  > and provide commented, minimal, self-contained, reproducible code.
>
>
> --
> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
> Biology, UCT), Dipl. Phys. (Germany)
>
> Centre of Excellence for Invasion Biology
> Stellenbosch University
> South Africa
>
> Tel : +33 - (0)9 53 10 27 44
> Cell: +33 - (0)6 85 62 59 98
> Fax : +33 - (0)9 58 10 27 44
>
> Fax (D): +49 - (0)3 21 21 25 22 44
>
> email: Rainer at krugs.de <mailto:Rainer at krugs.de>
>
> Skype: RMkrug
>
>


-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation 
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer at krugs.de

Skype:      RMkrug