Assumptions
The workbooks and a pdf version of this guide can be downloaded from here.
In every meta-analysis the following assumptions should be made, and the researcher is supposed to have verified that they are true for the meta-analysis at hand:
1. An effect is precisely defined, i.e., an independent as well as a dependent variable are defined, and all studies in the meta-analysis are empirical studies of that effect. These definitions should be precise enough to allow the researcher to include (and exclude) studies on transparent grounds.
Note: This might seem to be an obvious assumption, but it occurs quite often that authors claim that they have studied an effect of some independent variable on a dependent variable whereas on closer inspection it appears that they have studied other variables and, hence, another effect.
2. The type of unit or object in which this effect might occur is specified (e.g., persons, countries, teams specific types of organisational units) and the domain for which the effect will be meta-analysed is clearly delimited (e.g., all persons, not all persons but only adults, or only women; all countries, not all countries but only developed countries; all teams, not all teams but only product development teams in specific industries; all marketing departments, not all marketing departments but only marketing departments in a specific economic sector).
3. Assuming that the researcher’s aim is to synthesize empirical results about the effect in a domain (e.g., all patients in the world who might benefit from a specific treatment), all empirical studies of the effect in that domain should have been identified.
Note: This is a problematic assumption. Usually the set of studies that is meta-analysed is not complete because some studies have not been published, or have been published in a form to which the researcher has no access, or have been published in a language that the researcher cannot read, etcetera.
4. All studies are methodologically sound, i.e., data have been collected from a complete probability sample of a defined population, measurement has been valid and reliable, and the statistical analysis has been adequate.
Note: This is also a problematic assumption because most studies fail one or more of these criteria: the population might not have been specified, probability sampling might not have been conducted, there might be missing cases, measurement might not be valid or reliable, statistical procedures might be inappropriate (e.g., when statistical methods for differences between independent groups are used in a pretest - posttest design). Note that verification of this assumption requires a methodological evaluation of each study, irrespective of its source or reputation (“peer-reviewed”, “highly cited”, “good journal”, etc.). If this quality requirement is neglected or violated, then any meta-analytic result is meaningless (garbage in, garbage out).
5. Effect size measures in these studies are comparable. Specifically, they need to have the same scale across studies.
Assumptions 1, 2, 4 and 5 will not be further discussed in this document. In other words, it is assumed throughout this text that the researcher has, as an input for meta-analysis, a set of comparable and methodologically sound effect sizes for specified populations from a domain. Assumption 3 will be further discussed on several occasions in the different sections of this guide.
