Tải bản đầy đủ
4 Practical Matters: The Reciprocal Relation between Planning and Conducting a Meta‑Analysis
PLANNING AND PREPARING A META-ANALYTIC REVIEW
has read empirical studies in the area that would likely be included in the
review, and conclusions that the reader takes from these studies will undoubtedly influence the type of questions asked when planning the meta-analysis.
Beyond this obvious example, I think that much of the process of conducting a meta-analysis is less linear than is typically presented, but more
of an iterative, back-and-forth process among the various steps of planning,
searching the literature, coding studies, analyzing the data, and writing the
results. I do not view this reality as problematic; although we should avoid
the practice of “HARKing” (Hypothesizing After Results are Known; Kerr,
1998), we do learn a lot during the process of conducting the meta-analysis
that can refine our initial questions. Next, I briefly describe how each of the
major steps of searching the literature, coding studies, analyzing the data,
and writing the results can provide reasons to revise our initial plans of the
As I discuss in detail in Chapter 3, an important step in meta-analysis
is specifying inclusion/exclusion criteria (i.e., what type of studies will be
included in the literature) and searching for relevant literature. This process should be guided by the research questions you wish to answer, but
the process might also change your research questions. For example, finding
that there is little relevant literature to inform your meta-analysis research
questions—either too few studies to obtain a good estimate of the overall
effect size or too little variation over levels of moderators of interest—might
force you to broaden your questions to include more studies. Conversely,
finding that so many studies are relevant to your research question that it is
not practical to include all of them might cause you to narrow your research
question (e.g., to a more limited sample, type of measure, and/or type of
Research questions can also be modified after you begin coding studies
(see Chapters 4–7). Not only might your careful reading of the studies lead
you to new or modified research questions, but also the more formal process
of coding might necessitate changes in your research questions. If studies
do not provide sufficient information to compute effect sizes consistently,
and it is not possible to obtain this information from study authors, then it
may be necessary to abandon or modify your original research questions. If
your research questions involve comparing studies (i.e., moderator analyses),
you may have to alter this research question if the studies do not provide
adequate variability or coverage of certain characteristics. For example, if
you were interested in evaluating whether an effect size differs across ethnic
groups, but during the coding of studies found that most studies sampled
only a particular ethnic group, then you would not have adequate variability
Questions That Can and Cannot Be Answered through Meta-Analysis 31
across the studies and would have to abandon this particular research question (or else modify it in some way to make it more tractable).
Analyzing the data (see Chapters 8–12) is probably where the most modifications to original study questions will occur. Although you should thoroughly investigate your original research questions, and you should avoid
entirely exploratory “fishing expeditions,” you will invariably form new
research questions during the data analysis phase. Some of these new questions will be formed as you learn answers to your original questions (e.g.,
“Having found this, I wonder if . . . ?”), whereas other questions will come
from simply looking at the data (e.g., thinking about why a particular study,
or set of studies, has discrepant effect sizes). Although both approaches are
post hoc, the latter is certainly more exploratory—and therefore more likely
to capitalize on chance—than the former. However, both approaches to creating new research questions are valuable, as long as you are upfront about
their source when presenting and drawing conclusions from your metaanalysis (see Chapter 13).
As is true of analyzing the data, the process of writing your results may
lead to refinement of research questions or even the development of new
ones. Furthermore, the process of presenting your findings to colleagues—
through either conference presentations or the peer review process—is likely
to generate further refinement and creation of research questions.
As with any research endeavor, it is important to identify the research questions you wish to answer when you are planning your meta-analysis. To facilitate generating and shaping these questions, I have described the primary
methods of meta-analysis as combination and comparison across studies,
with the focus being on one of a variety of effect sizes. I have also compared
potential limitations of primary research and meta-analysis to offer perspective on the ways that meta-analysis can (and cannot) improve upon existing
primary studies. In addition, I have discussed some of the common criticisms
of meta-analysis; although most of these are either inaccurate or else applicable to both meta-analytic and narrative reviews, early recognition of these
criticisms can help you avoid some of these charges. Finally, I have described
how formulating research questions for meta-analysis is a reciprocal process.
Although you should identify research questions during the planning stage,
it is likely that these will be modified or appended throughout the process of
conducting a meta-analysis.
PLANNING AND PREPARING A META-ANALYTIC REVIEW
Hall, J. A., Tickle-Degnen, L., Rosenthal, R., & Mosteller, F. (1994). Hypotheses and problems in research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of
research synthesis (pp. 17–28). New York: Russell Sage Foundation.—This chapter
provides a brief overview of some of the critical considerations in formulating problems
for a meta-analytic review.
Pan, M. L. (2008). Preparing literature reviews: Qualitative and quantitative approaches
(3rd ed.). Glendale, CA: Pyrczak.—This very accessible and brief book provides
pragmatic advice for students preparing a literature review. Chapter 2 describes ways
to identify potential topics for review. The book as a whole is probably most useful for
undergraduate or early graduate students.
1. Rosenthal (1991) identified a third general approach called aggregate analysis.
This approach evaluates some characteristics of the studies in relation to some
mean value. For example, Rosenthal (1991) cited an analysis by Underwood
(1957) evaluating a methodological feature of 14 learning studies (number of
lists participants had to learn prior to the list of interest) with the mean values
of participants in those 14 studies (mean amount of material recalled from the
list of interest). This type of aggregate analysis represents a special case of metaanalysis in which the effect size is a single variable (in this example, a mean; see
Chapter 7, Section 7.1) rather than the more typical effect size representing an
association between two variables (see Chapter 5). Associations of this single
variable with study characteristics represent what I have described as moderator analyses in Chapter 9 (though the term “moderator analysis” is not accurate
when the effect size is a single variable, these same techniques apply). Given that
the aggregate analysis described by Rosenthal (1991) represents a special case of
the more general meta-analytic approach I describe in this book, I do not consider it a third, unique goal of meta-analysis.
2. The single exception to this statement is that primary studies that have overly
heterogeneous samples (e.g., sampling extreme cases) can suffer expansion of
range of variables of interest, which leads to overestimation of effect sizes.
3. Strictly speaking, the researcher can modify each of these factors, though it is
typically difficult to do so. Type I error rates are arbitrary, yet most fields have
such entrenched standards (most often a = .05) that deviations are met with skepticism. The researcher is of course free to choose from a range of data-analytic
strategies, but generally there is little variability in this decision because all
researchers will choose the approach that provides the highest statistical power.
Finally, special design limitations, such as intentionally sampling homogeneous
Questions That Can and Cannot Be Answered through Meta-Analysis 33
populations to reduce extraneous variance in between-group comparisons, can
impact the population effect size; but this effect size is generally considered to
exist independent of the researcher’s control.
4. Two important caveats to this claim merit consideration. First, it is necessary for
the collection of underpowered studies to have enough methodological similarities if they are to be considered reasonable replications of one another (Halpern
et al., 2002). Second, some underpowered studies will still yield statistically significant results because they, by chance alone, happen to estimate a particularly
large effect size that achieved statistical significance, whereas other underpowered studies will yield more typical effect sizes that fail to achieve statistical significance. If the former are more likely to be published (or otherwise included in
your meta-analysis) than the latter, then meta-analytic combination of this biased
sample of underpowered effect sizes will lead to overestimates of the effect size
in your meta-analysis. This possibility has led to suggestions that you exclude
underpowered studies from meta-analyses (Kraemer, Gardner, Brooks, & Yesavage, 1998). Although I view the categorical exclusion of all underpowered studies
as being problematic in areas of research where there are few adequately powered
studies, the possibility of underpowered studies being systematically biased does
indicate the importance of thorough literature reviews (especially for unpublished studies; see Chapter 3), the value of conducting analyses to detect publication bias (see Chapter 11), and the need for caution in interpreting meta-analyses
of underpowered studies.
5. A meta-analysis using fixed-effects analysis (see Chapter 8) will always have
greater statistical power in determining the mean effect size than will any one
of the multiple studies included. Meta-analyses using random-effects models
(Chapter 9) can have lower statistical power than a single primary study if substantial population heterogeneity in effect sizes exists.
6. I should make clear that basic meta-analytic techniques require only this level of
prerequisite knowledge, but there are some more advanced meta-analytic techniques that require understanding of matrix algebra and multivariate statistics.
Most of the material in this book is accessible for readers with basic graduatelevel training, though I also include—with appropriate warning—some of this
more advanced material.
7. In principle, it is acceptable to identify a large sample of studies and randomly
sample a more tractable number of studies. In practice, however, nearly all metaanalyses that are published include all identified studies in the coding and analyses, which means that many readers expect you to include all available studies in
Searching the Literature
After articulating one or more research questions for your meta-analysis (Chapter 2), the next step is to locate the studies that will provide information to
answer these questions (as described in subsequent chapters on coding and
analysis). Unlike narrative reviews that are typically unsystematic in their searching of the literature (or at least typically do not articulate this process), the field
of meta-analysis has devoted considerable attention to practices of searching
and retrieving relevant literature.
In this chapter, I describe how it is useful to conceptualize the studies in
your meta-analysis as a sample of a larger population (Section 3.1) and how
this conceptualization leads to explicit criteria of which type of studies should
be included versus excluded from your meta-analysis (Section 3.2). I will then
describe various methods of searching for relevant literature, considering the
advantages and disadvantages of each (Section 3.3). I conclude the chapter
by describing the importance of “reality checking” your search (Section 3.4)
and the practical matter of creating a meta-analytic database (Section 3.5).
The steps involved in a literature search as described in this chapter are summarized in Figure 3.1.
3.1Developing and Articulating
a Sampling Frame
Given that meta-analysis uses the individual study as its unit of analysis, it is
useful to think of your meta-analysis as consisting of a sample of studies, just
as primary analyses sample people or other units (e.g., families, businesses)
6. Initial list of studies