Conclusion

Meta-analysis is a statistical tool that supports a synthesis and evaluation of the results of studies about an effect that have been conducted and published, and that after publication have been retrieved, read, and critically evaluated by the researcher. All results of the meta-analysis pertain to the set of results that happen to be generated and retrieved. These results are subject to an unknown degree of selection bias which inevitably results from the arbitrary way in which populations (or “samples”) are selected for the separate studies. Often, some of the more glaring biases can be known, e.g., when a research literature mainly consists of results of experiments with students and has not covered effects in “real life”. Therefore, the first part of any interpretation of meta-analytic results should be an explicit statement about populations that are not yet covered by empirical research and in which different effects might have been observed. Recommendations might be formulated for further research in yet not researched populations.

Meta-analysis generates estimates of a (weighted) average effect size, of the dispersion of effect sizes, of the homogeneity (or heterogeneity) of the total set of observed effect sizes and of subgroups, and supports the exploration of the relevance of potential moderators. Before conclusions are drawn, the degree of heterogeneity should first be assessed and interpreted. “Combined” effect sizes should only be used as an outcome if homogeneity of a group or subgroup of observed effect sizes is without doubt and, even then, only for the domain that is defined by this specific group of populations.

Because relevant heterogeneity is normally found in the social sciences, the main result of most meta-analyses is an insight in the dispersion of true effects. In those cases, meta-analysis functions as a tool for generating hypotheses about “moderators” of the effect.

Meta-analysis should not be used for “testing”, and it should not be used for generating statements about the size of an effect in not yet researched parts of the domain or in the entire domain.

In sum, conclusions from a meta-analysis may take the following form:

• Studies of this effect (X → Y) have been conducted in the following populations. The following (types of) populations have not been studied.
• Observed effects range from … to ….
• Effects in subgroup A … range from … to … This means that if X increases with Δx in a population of this subgroup, then Y might increase with at least Δy.
• Effects in subgroup B … range from … to … This means that if X increases with Δx in a population of this subgroup, then Y might increase with at least Δy.