[R] Stack smashing again, redux.

[Ivan Krylov]

Depending on one's dislike for different approaches, it's possible to
use AddressSanitizer with R in at least three different ways, probably
more. There's the Rocker project providing Docker images of R already
built with sanitizer support [1] (but then you have to install Docker),
there's compiling R from source with -fsanitize=address in CFLAGS,
FFLAGS, MAIN_LDFLAGS [2] (but then you have to compile R from source)
and there's the partially manual way I've mentioned before (which
involves modifying one's global configuration files and reverting the
changes later):

1. Temporarily add the following to ~/.R/Makevars:
   FFLAGS=-g -Og -fsanitize=address
   FCFLAGS=-g -Og -fsanitize=address
2. Compile the shared object using:
   R CMD SHLIB -o hah.so *.f -fsanitize=address
3. Run R as follows:
   LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.5 R

With AddressSanitizer, I get the following stack buffer overflow error
message:

> xxx <- get.gl(theta.new,sigma,X,y,cf,state,"Dbd",size,nbot,ntop)
=================================================================
==716==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffea5d32458 at pc 0x7f282c38498d bp 0x7ffea5d32
000 sp 0x7ffea5d31ff8
WRITE of size 8 at 0x7ffea5d32458 thread T0
    #0 0x7f282c38498c in derivfdbd_ /home/ivan/derivfdbd.f:16
    #1 0x7f282c3852d6 in derivf_ /home/ivan/derivf.f:17
    #2 0x7f282c385dd3 in getgl_ /home/ivan/getgl.f:19
    #3 0x7f282c4e5eec in do_dotCode src/main/dotcode.c:1994

(skipping unrelated stack frames)

Address 0x7ffea5d32458 is located in stack of thread T0 at offset 312 in frame
    #0 0x7f282c385a6f in getgl_ /home/ivan/getgl.f:2

  This frame has 7 object(s):
    [32, 36) 'nd'
    [96, 104) 'd2aa'
    [160, 168) 'd2ab'
    [224, 232) 'd2bb'
    [288, 296) 'd2f' <== Memory access at offset 312 overflows this variable
    [352, 360) 'd2u'
    [416, 424) 'd2zeta'

(skipping more unnecessary information)

Adding "dimension d2f(kstate,npar,npar)" to getgl.f seems to prevent
this or any other error from happening, though I can't judge the
calculation results; they could indicate some other problem with memory
management.

-- 
Best regards,
Ivan

[1] https://www.rocker-project.org/images/#additional-images

[2]
https://cran.r-project.org/doc/manuals/R-exts.html#Using-Address-Sanitizer