[R] metafor package: effect sizes are not fully independent

[Gang Chen]

Thanks for the confirmation and pointer, Mike!

Dr. Viechtbauer, I'm looking forward to the new functionality of
specifying covariance structures in metafor().

Thanks both again for the great help,
Gang


On Sun, Feb 7, 2010 at 8:59 PM, Mike Cheung <mikewlcheung at gmail.com> wrote:
> Dear Gang,
>
> It seems that it is possible to use a univariate meta-analysis to
> handle your multivariate effect sizes. If you want to calculate a
> weighted average first, Hedges and Olkin (1985) has discussed this
> approach.
>
> Hedges, L. V., & Olkin, I. (1985). Statistical methods for
> meta-analysis. Orlando, FL: Academic Press.
>
> Regards,
> Mike
> --
> ---------------------------------------------------------------------
>  Mike W.L. Cheung               Phone: (65) 6516-3702
>  Department of Psychology       Fax:   (65) 6773-1843
>  National University of Singapore
>  http://courses.nus.edu.sg/course/psycwlm/internet/
> ---------------------------------------------------------------------
>
> On Mon, Feb 8, 2010 at 6:48 AM, Gang Chen <gangchen6 at gmail.com> wrote:
>> Dear Mike,
>>
>> Thanks a lot for the kind help!
>>
>> Actually a few months ago I happened to read a couple of your posts on
>> the R-help archive when I was exploring the possibility of using lme()
>> in R for meta analysis.
>>
>> First of all, I didn't specify the meta analysis model for my cases
>> correctly in my previous message. Currently I'm only interested in
>> random- or mixed-effects meta analysis. So what you've suggested is
>> directly relevant to what I've been looking for, especially for case
>> (2). I'll try to gather those references you listed, and figure out
>> the details.
>>
>> Also I think I didn't state my case (1) clearly in my previous post.
>> In that case, all the effect sizes are the same and in the same
>> condition too (e.g., happy), but each source has multiple samples of
>> the measurement (and also measurement error, or standard error). Could
>> this still be handled as a multivariate meta analysis since the
>> samples for the the same source are correlated? Or somehow the
>> multiple measures from the same source can be somehow summarized
>> (weighted average?) before the meta analysis?
>>
>> Your suggestions are highly appreciated.
>>
>> Best wishes,
>> Gang
>>
>>
>> On Sun, Feb 7, 2010 at 10:39 AM, Mike Cheung <mikewlcheung at gmail.com> wrote:
>>> Dear Gang,
>>>
>>> Here are just some general thoughts. Wolfgang Viechtbauer will be a
>>> better position to answer questions related to metafor.
>>>
>>> For multivariate effect sizes, we first have to estimate the
>>> asymptotic sampling covariance matrix among the effect sizes. Formulas
>>> for some common effect sizes are provided by Gleser and Olkin (2009).
>>>
>>> If a fixed-effects model is required, it is quite easy to write your
>>> own GLS function to conduct the multivariate meta-analysis (see e.g.,
>>> Becker, 1992). If a random-effects model is required, it is more
>>> challenging in R. SAS Proc MIXED can do the work (e.g., van
>>> Houwelingen, Arends, & Stijnen, 2002).
>>>
>>> Sometimes, it is possible to transform the multivariate effect sizes
>>> into independent effect sizes (Kalaian & Raudenbush, 1996; Raudenbush,
>>> Becker, & Kalaian, 1988). Then univariate meta-analysis, e.g.,
>>> metafor(), can be performed on the transformed effect sizes. This
>>> approach works if it makes sense to pool the multivariate effect sizes
>>> as in your case (2)- the effect sizes are the same but in different
>>> conditions (happy, sad, and neutral). However, this approach does not
>>> work if the multivariate effect sizes are measuring different
>>> concepts, e.g., verbal achievement and mathematical achievement.
>>>
>>> Hope this helps.
>>>
>>> Becker, B. J. (1992). Using results from replicated studies to
>>> estimate linear models. Journal of Educational Statistics, 17,
>>> 341-362.
>>> Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect
>>> sizes. In H. Cooper, L. V. Hedges, and J. C. Valentine (Eds.), The
>>> handbook of research synthesis and meta-analysis, 2nd edition (pp.
>>> 357-376). New York: Russell Sage Foundation.
>>> Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed
>>> linear model for meta-analysis. Psychological Methods, 1, 227-235.
>>> Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling
>>> multivariate effect sizes. Psychological Bulletin, 103, 111-120.
>>> van Houwelingen, H.C., Arends, L.R., & Stijnen, T. (2002). Advanced
>>> methods in meta-analysis: multivariate approach and meta-regression.
>>> Statistics in Medicine, 21, 589-624.
>>>
>>> Regards,
>>> Mike
>>> --
>>> ---------------------------------------------------------------------
>>>  Mike W.L. Cheung               Phone: (65) 6516-3702
>>>  Department of Psychology       Fax:   (65) 6773-1843
>>>  National University of Singapore
>>>  http://courses.nus.edu.sg/course/psycwlm/internet/
>>> ---------------------------------------------------------------------
>>>
>>> On Sat, Feb 6, 2010 at 6:07 AM, Gang Chen <gangchen6 at gmail.com> wrote:
>>>> In a classical meta analysis model y_i = X_i * beta_i + e_i, data
>>>> {y_i} are assumed to be independent effect sizes. However, I'm
>>>> encountering the following two scenarios:
>>>>
>>>> (1) Each source has multiple effect sizes, thus {y_i} are not fully
>>>> independent with each other.
>>>> (2) Each source has multiple effect sizes, and each of the effect size
>>>> from a source can be categorized as one of a factor levels (e.g.,
>>>> happy, sad, and neutral). Maybe better denote the data as y_ij, effect
>>>> size at the j-th level from the i-th source. I can code the levels
>>>> with dummy variables into the X_i matrix, but apparently the data from
>>>> the same source are correlated with each other. In this case, I would
>>>> like to run a few tests one of which is, for example, whether there is
>>>> any difference across all the levels of the factor.
>>>>
>>>> Can metafor handle these two cases?
>>>>
>>>> Thanks,
>>>> Gang
>>
>