Tải bản đầy đủ
1 Meta‑analysis to Obtain Sufficient Statistics

# 1 Meta‑analysis to Obtain Sufficient Statistics

Tải bản đầy đủ

Multivariate Meta-­Analytic Models

281

variables (a square matrix, Rii, with the number of rows and columns equal
to the number of predictors), and (2) the correlations of each independent
variable with the dependent variable (a column vector, Riy, with the number
of rows equal to the number of predictors, using the following equations1
(Tabachnick & Fidell, 1996, p. 142):
Equation 12.1: Matrix equations for multiple regression

B i  R ii
1R iy
R 2  R `iy B i

• Bi is a k × 1 vector of standardized regression coefficients.
• Rii is a k × k matrix of correlations among independent variables.
• Riy is a k × 1 vector of correlations of independent variables with
the dependent variable.
• R2 is the proportion of variance in the dependent variable predicted by the set of independent variables.
• k is the number of predictors.

12.1.1.b Exploratory Factor Analysis
Exploratory factor analysis (EFA) is used to extract a parsimonious set of factors that explain associations among a larger set of variables. This approach
is commonly used to determine (1) how many factors account for the associations among variables, (2) the strengths of associations of each variable on
a factor (i.e., the factor loadings), and (3) the associations among the factors
(assuming oblique rotation). For each of these goals, exploratory factor analysis is preferred to principal components analysis (PCA; see, e.g., Widaman,
1993, 2007), so I describe EFA only. I should note that my description here is
brief and does not delve into the many complexities of EFA; I am being brief
because I seek only to remind you of the basic steps of EFA without providing
a complete overview (for more complete coverage, see Cudeck & MacCallum,
2007).
Although the matrix algebra of EFA can be a little daunting, all that is
initially required is the correlation matrix (R) among the variables, which is
a square matrix of p rows and columns (where p is the number of variables).
From this correlation matrix, it is possible to compute a matrix of eigenvectors, V, which has p rows and m columns (where m is the number of factors).2

282

COMBINING AND COMPARING EFFECT SIZES

To determine the number of factors that can be extracted, you extract the
maximum number of factors3 and then examine the resulting eigenvalues
contained in the diagonal matrix (m × m) L:
Equation 12.2: Matrix equations for computing eigenvalues
from EFA factor extraction

L = V′RV
• L is a m × m diagonal matrix of eigenvalues.
• V is a p × m matrix of eigenvectors.
• R is a p × p matrix of correlations among variables.
• p is the number of variables.
• m is the number of factors.

You decide on the number of factors to retain based on the magnitudes
of the eigenvalues contained in L. A minimum (i.e., necessary but not sufficient) threshold is known as Kaiser’s (1970) criterion, which states that the
eigenvalue is greater than 1.0. Beyond this criterion, it is common to rely on a
scree plot, sometimes with parallel analysis, as well as considering the interpretability of rival solutions, to reach a final determination of the number of
factors to retain.
The analysis then proceeds with a specified number of factors (i.e.,
some fixed value of m that is less than p). Here, the correlation matrix (R)
is expressed in terms of a matrix of unrotated factor loadings (A), which are
themselves calculated from the matrices of eigenvectors (V) and eigenvalues
(L):
Equation 12.3: Matrix equations for computing unrotated factor

R = AA′
AV L
• R is a p × p matrix of correlations among variables.
• V is a p × m matrix of eigenvectors.
• L is a m × m diagonal matrix of eigenvalues.
• p is the number of variables.
• m is the number of factors.

Multivariate Meta-­Analytic Models

283

the matrix A), you typically apply a rotation of some sort. Numerous rotations exist, with the major distinction being between orthogonal rotations,
in which the correlations among factors are constrained to be zero, versus
oblique rotations, in which nonzero correlations among factors are estimated.
Oblique rotations are generally preferable, given that it is rare in social sciences for factors to be truly orthogonal. However, oblique rotations are also
more computationally intensive (though this is rarely problematic with modern computers) and can yield various solutions using different criteria, given
that you are attempting to estimate both factor loadings and factor intercorrelations simultaneously. I avoid the extensive consideration of alternative
estimation procedures by simply stating that the goal of each approach is to
produce a reproduced (i.e., model implied) correlation matrix that closely
corresponds (by some criterion) to the actual correlation matrix (R). This
reproduced matrix is a function of (1) the pattern matrix (A), which here
(with oblique rotation) represents the unique relations of variable with factors (controlling for associations among factors), and (2) the factor correlation matrix (F), which represents the correlations among the factors4:
Equation 12.4: Matrix equation for reproduced correlation
matrix in EFA

ˆ = AΦA′
R
ˆ is a p × p matrix of model-­implied correlations among vari•R
ables.
• A is a p × m matrix of unique associations between variables and
factors (controlling for associations among factors).
• F is a m × m matrix of correlations among factors.
• p is the number of variables.
• m is the number of factors.

When the reproduced correlation matrix (Rˆ ) adequately reproduces the
observed correlation matrix (R), the analysis is completed. You then interpret
the values within the pattern matrix (A) and matrix of factor correlations (F)
to address the second and third goals of EFA described above.

284

COMBINING AND COMPARING EFFECT SIZES

12.1.1.c

Confirmatory Factor Analysis

In many cases, it may be more appropriate to rely on a confirmatory, rather
than an exploratory, factor analysis. A confirmatory factor analysis (CFA) is
estimated by fitting the data to a specified model in which some factor loadings (or other parameters, such as residual covariances among variables) are
specified as fixed to zero versus freely estimated. Such a model is often a more
realistic representation of your expected factor structure than is the EFA.5
Like the EFA, the CFA estimates associations among factors (typically
called “constructs” or “latent variables” in CFA) as well as strengths of associations between variables (often called “indicators” or “manifest variables”
in CFA) and constructs. These parameters are estimated as part of the general
CFA matrix equation6:
Equation 12.5: Matrix equation for CFA

S = ΛΨΛ′ + Θ
• S is a p × p matrix of model-­implied variances and covariances
among manifest variables.
• Λ is a p × m matrix of factor loadings of manifest variables regressed
onto constructs.
• Ψ is a m × m matrix of variances and covariances among constructs (latent variables).
• Θ is a p × p matrix of residual variances and covariances among
manifest variables.
• p is the number of manifest variables.
• m is the number of constructs (latent variables).

To estimate a CFA, you place certain constraints on the model to set the
scale of latent constructs (see Little, Slegers, & Card, 2006) and ensure identification (see Kline, 2010, Ch. 6). For example, you might specify that there
is no factor loading of a particular indicator on a particular construct (vs. an
EFA, in which this would be estimated even if you expected the value to be
small). Using Equation 12.5, a software program (e.g., Lisrel, EQS, Mplus) is
used to compute values of factor loadings (values within the Λ matrix), latent
variances and covariances (values within the Ψ matrix), and residual variances (and sometimes residual covariances; values within the Θ matrix) that
yield a model implied variance/covariance matrix, S. The values are selected
so that this model-­implied matrix closely matches the observed (i.e., from the

Multivariate Meta-­Analytic Models

285

data) variances and covariance matrix (S) according to some criterion (most
commonly, the maximum likelihood criterion minimizing a fit function). For
CFA of primary data, the sufficient statistics are therefore the variances and
covariances comprising S; however, it is also possible to use correlation coefficients such as would be available from meta-­analysis to fit CFAs (see Kline,
2010, Ch. 7).7
12.1.2The Logic of Meta‑Analytically Deriving
Sufficient Statistics
The purpose of the previous section was not to fully describe the matrix
equations of multiple regression, EFA, and CFA. Instead, I simply wish to
illustrate that a range of multivariate analyses can be performed using only
correlations. Other multivariate analyses are possible, including canonical
correlations, multivariate analysis of variance or covariance, and structural
equation modeling. In short, any analysis that can be performed using a correlation matrix as sufficient information can be used as a multivariate model
for meta-­analysis.
The “key” of multivariate meta-­analysis then is to use the techniques
of meta-­analysis described throughout this book to obtain average correlations from multiple studies. Your goal is to compute a meta-­analytic mean
correlation for each of the correlations in a matrix of p variables. Therefore,
your task in a multivariate meta-­analysis is not simply to perform one meta­analysis to obtain one correlation, but to perform multiple meta-­analyses
to obtain all possible correlations among a set of variables. Specifically, the
number of correlations in a matrix of p variables is equal to p(p –1)/2. This
correlation matrix (R) of these mean correlations is then used in one of the
multivariate analyses described above.
12.1.3The Challenges of Using Meta‑Analytically
Deriving Sufficient Statistics
Although the logic of this approach is straightforward, several complications
arise (see Cheung & Chan, 2005a). The first is that it is unlikely that every
study that provides information on one correlation will provide information
on all correlations in the matrix. Consider a simple situation in which you
wish to perform some multivariate analysis of variables X, Y, and Z. Study
1 might provide all three correlations (rXY, rXZ, and rYZ). However, Study 2
did not measure Z, so it only provides one correlation (rXY); Study 3 failed to
measure Y and so also provides only one correlation (rXZ); and so on. In other

286

COMBINING AND COMPARING EFFECT SIZES

words, multivariate meta-­analysis will almost always derive different average
correlations from different subsets of studies.
This situation poses two problems. First, it is possible that different correlations from very different sets of studies could yield a correlation matrix
that is nonpositive definite. For example, imagine that three studies reporting rXY yield an average value of .80 and four studies reporting rXZ yield an
average value of .70. However, the correlation between Y and Z is reported in
three different studies, and the meta-­analytic average is –.50. It is not logically possible for there to exist, within the population, a strong positive correlation between X and Y, a strong positive correlation between X and Z, but
a strong negative correlation between Y and Z.8 Most multivariate analyses
cannot use such nonpositive definite matrices. Therefore, the possibility that
such nonpositive definite matrices can occur if different subsets of studies
inform different correlations within the matrix represents a challenge to
multivariate meta-­analysis.
Another challenge that arises from the meta-­analytic combination of
different studies for different correlations within the matrix has to do with
uncertainty about the effective sample size. Although many multivariate
analyses can provide parameter estimates from correlations alone, the standard errors of these estimates (for significance testing or constructing confidence intervals) require knowledge of the sample size. When the correlations
are meta-­analytically combined from different subsets of studies, it is unclear
what sample size should be used (e.g., the smallest sum of participants among
studies for one of the correlations; the largest sum; or some average?).
A final challenge of multivariate meta-­analysis is how we manage heterogeneity among studies. By computing a matrix of average correlations, we are
implicitly assuming that one value adequately represents the populations of
effect sizes. However, as I discussed earlier, it is more appropriate to test this
homogeneity (vs. heterogeneity; see Chapter 8) and to model this population
heterogeneity in a random-­effects model if it exists (see Chapter 9). Only one
of the two approaches I describe next can model between-study variances in
a random-­effects model.

12.2Two Approaches to Multivariate
Meta‑Analysis
Given the challenges I described in the previous section, multivariate meta­analysis is considerably more complex than simply synthesizing several correlations to serve as input for a multivariate analysis. The development of models

Multivariate Meta-­Analytic Models

287

that can manage these challenges is an active area of research, and the field has
currently not resolved which approach is best. In this section, I describe two
approaches that have received the most attention: the meta-­analytic structural
equation modeling (MASEM) approach describe by Cheung and Chan (2005a)
and the generalized least squares (GLS) approach by Becker (e.g., 2009). I
describe both for two reasons. First, you might read meta-­analyses using either
approach, so it is useful to be familiar with both. Second, given that research
on both approaches is active, it is difficult for me to predict which approach
might emerge as superior (or, more likely, superior in certain situations). However, as the state of the field currently stands, the GLS approach is more flexible in that it can estimate either fixed- or random-­effects mean correlations
(whereas the MASEM approach is limited to fixed-­effects models9). For this
reason, I provide considerably greater coverage of the GLS approach.
To illustrate these approaches, I expand on the example described earlier in the book. Table 12.1 summarizes 38 studies that provide correlations
among relational aggression (e.g., gossiping), overt aggression (e.g., hitting),
and peer rejection.10 Here, 16 studies provide all three correlations among
these variables, 6 provide correlations of both relational and overt aggression
to peer rejection, and 16 provide the correlation between overt and relational
aggression. This particular example is somewhat artificial, in that (1) a selection criterion for studies in this review was that results be reported for both
relational and overt forms of aggression (otherwise, there would not be perfect overlap in the correlations of these two forms with peer rejection), and
(2) for simplicity of presentation, I selected only the first 16 studies, out of
82 studies in the full meta-­analysis, that provided only the overt with relational aggression correlation. Nevertheless, the example is realistic in that
the three correlations come from different subsets of studies, and contain
different numbers of studies and participants (for rrelational-overt, k = 32, N =
11,642; for rrelational-­rejection and rovert-­rejection, k = 22, N = 8,081). I next use
this example to illustrate how each approach would be used to fit a multiple
regression of both forms of aggression predicting peer rejection.
12.2.1The MASEM Approach
One broad approach to multivariate meta-­analysis is the MASEM approach
described by Cheung and Chan (2005a). This approach relies on SEM methodology, so you must be familiar with this technique to use this approach.
Given this restriction, I write this section with the assumption that you are at
least somewhat familiar with SEM (if you are not, I highly recommend Kline,
2010, as an accessible introduction).

288

COMBINING AND COMPARING EFFECT SIZES

TABLE 12.1. Example Multivariate Meta-­Analysis of Correlations
among Relational Aggression, Overt Aggression, and Peer Rejection
Study
Andreou (2006)
Arnold (1998)
Berdugo-­Arstark (2002)
Blachman (2003)
Brendgen et al. (2005)
Butovskaya et al. (2007)
Campbell (1999)
Carpenter (2002)
Carpenter & Nangle (2006)
Cillessen & Mayeux (2004)
Cillessen et al. (2005)
Côté et al. (2007)
Coyne & Archer (2005)
Crain (2002)
Crick (1995)
Crick (1996)
Crick (1997)
Crick & Grotpeter (1995)
Crick et al. (1997)
Geiger (2003)
Hawley et al. (2007)
Henington (1996)
Johnson (2003)
Leff (1995)
Miller (2001)
Murray-Close & Crick (2006)
Nelson et al. (2005)
Ostrov (under review)a
Ostrov & Crick (2007)
Ostrov et al. (2004)
Pakaslahti & Keltikangas-Järvinen (1998)
Phillipsen et al. (1999)
Rys & Bear (1997)
Salmivalli et al. (2000)
Werner (2000)
Werner & Crick (2004)
Zalecki & Hinshaw (2004)

Sample
size (N)

Relational–
overt r

403
110
128
228
468
212
139
75
82
607
224
1183
347
134
252
245
1166
491
65
458
929
904
74
151
150
590
180
139
132
60
839
262
266
209
314
881
517
228

.472
.707
.549
.440
.420
.576
.641
.260
.270
.561
.652
.345
.540
.870
.656
.770
.630
.540
.607
.650
.669
.561
.735
.790
.530
.700
.584
.403
.030
.580

.681
.666

.440

Relational–­
rejection r

Overt–­
rejection r

.483

.592

.121
.280
.520
.146
.310
.368
.570
.530
.540
.030
.332
.045
.000b
.269
–.045
.423
.240
.153
.440
.440
.516

.228
.367
.480
.089
.295
.527
.570
.420
.510
.304
.402
.155
.100
.250
.013
.378
.385
.240
.430
.430
.562

aArticle was under review during the preparation of this meta-­analytic review. It has subsequently been
published as Ostrov (2008).
bEffect size is lower-bound estimate based on author’s reporting only nonsignficant associations.

Multivariate Meta-­Analytic Models

289

In this approach, you treat the correlation matrix from each study as
sufficient statistics for a group in a multigroup SEM. In other words, each
study is treated as a group, and the correlations obtained from each study
are entered as the data for that group. Although the multigroup approach is
relatively straightforward if all studies provided all correlations, this is typically not the case. The MASEM approach accounts for situations in which
some studies do not include some variables, by not estimating the parameters
involving those variables for that “group.” However, the parameter estimates
are constrained equal across groups, so identification is ensured (assuming
that the overall model is identified). Note that this approach considers the
completeness of studies in terms of variables rather than correlations (in contrast to the GLS approach described in Section 12.2.2). In other words, this
approach assumes that if a variable is present in a study, the correlations of
that variable with all other variables in the study are present. To illustrate
using the example, if a study measured relation aggression, overt aggression,
and peer rejection, then this approach requires that you obtain all three correlations among these variables. If a study measured all three variables, but
failed to report the correlation between overt aggression and rejection (and
you could not obtain this correlation), then you would be forced to treat the
study as if it failed to measure either overt aggression or rejection (i.e., you
would ignore either the relational-overt or the relational-­rejection correlation).
The major challenge to this approach comes from the equality constraints
on all parameters across groups. These constraints necessarily imply that the
studies are homogeneous. For this reason, Cheung and Chan (2005a) recommended that the initial step in this approach be to evaluate the homogeneity versus heterogeneity of the correlation matrices. They propose a method
in which you evaluate heterogeneity through nested-model comparison of
an unrestricted model in which the correlations are freely estimated across
studies (groups) versus a restricted model in which they are constrained
equal.11 If the change is nonsignificant (i.e., the null hypothesis of homogeneity is retained), then you use the correlations (which are constrained
equal across studies) and their asymptotic covariance matrix as sufficient
statistics for your multivariate model (e.g., multiple regression in my example
or, as described by Cheung & Chan, 2005a, within an SEM). However, if the
change is significant (i.e., the alternate hypothesis of heterogeneity), then it
is not appropriate to leave the equality constraints in place. In this situation
of heterogeneity, this original MASEM approach cannot be used to evaluate
models for the entire set of studies (but see footnote 9). Cheung and Chan
(2005a) offer two recommendations to overcome this problem. First, you

290

COMBINING AND COMPARING EFFECT SIZES

might divide studies based on coded study characteristics until you achieve
within-group homogeneity. If you take this approach, then you must focus
on moderator analyses rather than make overall conclusions. Second, if the
coded study characteristics do not fully account for the heterogeneity, you
can perform the equivalent of a cluster analysis that will empirically classify studies into more homogeneous subgroups (Cheung & Chan, 2005b).
However, the model results from these multiple empirically identified groups
might be difficult to interpret.
Given the requirement of homogeneity of correlations, this approach
might be limited if your goal is to evaluate an overall model across studies.
In the illustrative example, I found significant heterogeneity (i.e., increase in
model misfit when equality constraints across studies were imposed). I suspect that this heterogeneity is likely more common than homogeneity. Furthermore, I was not able to remove this heterogeneity through coded study
characteristics. To use this approach, I would have needed to empirically
classify studies into more homogeneous subgroups (Cheung & Chan, 2005b);
however, I was dissatisfied with this approach because it would have provided
multiple sets of results without a clear conceptual explanation. Although this
MASEM approach might be modified in the future to accommodate heterogeneity (look especially for work by Mike Cheung), it currently did not fit
my needs within this illustrative meta-­analysis of relational aggression, overt
aggression, and peer rejection. As I show next, the GLS approach was more
tractable in this example, which illustrates its greater flexibility.
12.2.2The GLS Approach
Becker (1992; see 2009 for a comprehensive overview) has described a GLS
approach to multivariate meta-­analysis. This approach can be explained in
seven steps; I next summarize these steps as described in Becker (2009) and
provide results for the illustration of relational and overt aggression predicting peer rejection.
12.2.2.a Data Management
The first step is to arrange the data in a way that information from each
study is summarized in two matrices. The first matrix is a column vector of
the Fisher’s transformed correlations (Zr) from each study i, denoted as zi.
The number of rows of this matrix for each study will be equal to the number of correlations provided; for example, from the data in Table 12.1, this

Multivariate Meta-­Analytic Models

291

matrix will have one row for the Andreou (2006) study, three rows for the
Blachman (2003) study, and two rows for the Ostrov et al. (2004) study. The
second matrix for each study is an indicator matrix (Xi) that denotes which
correlations are represented in each study. The number of columns in this
matrix will be constant across studies (the total number of correlations in
the meta-­analysis), but the number of rows will be equal to the number of
correlations in the particular study. To illustrate these matrices, consider the
33rd study in Table 12.1, that by Rys and Bear (1997); the matrices (note that
the z matrix contains Fisher’s transformations of rs shown in the table) for
this study are:

§.451¶
§0 1 0¶
z 33  ¨
, X33  ¨
·
·

Note that this study, which provides two of the three correlations, is
represented with matrices of two rows. The indicator matrix (X 33) specifies
that these two correlations are the second and third correlations under consideration (the order is arbitrary, but needs to be consistent across studies;
here, I have followed the order shown in Table 12.1).
12.2.2.b Estimating Variances and Covariances of Study Effect
Size Estimates
Just as it was necessary to compute the standard errors of study effect size
estimates in all meta-­analyses (see Chapters 5 and 8), we must do so in this
approach to multivariate meta-­analysis. Here, I describe the variances of estimates of effect sizes, which is simply the standard error squared: Var(Zr) =
SEZr2. So the variances of each Zr effect size are simply 1 / (Ni – 3). However,
for a multivariate meta-­analysis, in which multiple effect sizes are considered, you must consider not only the variance of estimate of each effect size,
but also the covariances among these estimates (i.e., the uncertainty of estimation of one effect size is associated with the uncertainty of estimation of
another effect size within the same study). The covariance of the estimate
of the Fisher’s transformed correlation between variables s and t with the
estimate of the transformed correlation between variables u and v (where u
or v could equal s or t) from Study i is computed from the following equation
(Becker, 1992, p. 343; Beretvas & Furlow, 2006, p. 161)12: