Failsafe-N tests

The final part of the Publication Bias Analysis sheet contains several estimates of Failsafe numbers. To illustrate this, imagine that for any study, a number of other studies is not published. Assume that these additional studies have insignificant results, i.e. their effect sizes are essentially zero. Then, the failsafe number estimates the number of such additional studies that are required to turn the effect size from the included and additional studies combined insignificant, i.e. that the ‘new’ combined effect size is essentially zero.

Rosenthal

In order to calculate a Failsafe-N, first described by Rosenthal (1979), a test of combined significance is conducted. The failsafe number is the number of missing studies averaging a z-value of zero that should be added to make the combined effect size statistically insignificant (see Figure 26 for an example). The ad-hoc rule refers to the one by Rosenthal (1979) for deciding whether the number estimated is small (TRUE) or large (FALSE).

Gleser & Olkin

Gleser and Olkin (1996) provide an estimate for the number of unpublished results (see Figure 27 for an example). It uses the assumption that the studies in the meta-analysis have the largest significance (i.e., smallest p-values) from a population of effect sizes. The size of the largest p-value in the meta-analysis determines the number of estimated unpublished studies. There is no method to assess whether this number is small or large, but a comparison could be made with the number of studies that actually are included in the meta-analysis.

Orwin

Orwin (1983) uses a slightly different approach by looking at effect sizes rather than at p-values. For this method, the user sets a criterion value for the combined effect size. The user can set any value that would make the result of the meta-analysis arbitrary (ES_C) (see Figure 28 for an example). Secondly, the user sets the mean of the studies that are imputed (ES_FS). Then the failsafe number will give the number of studies with average effect size (ES_FS) that would reduce the combined effect to the criterion value (ES_C).

Fisher

The fourth and final failsafe number method provided by Meta-Essentials (proposed by Fisher, 1932) is also based on a test of the combined significance (see Figure 29 for an example). It is based on the sum of the natural logarithm of the p-values from the studies in the meta-analysis. The number can be tested with a Chi-Square distribution with degrees of freedom of two times the number of studies in the meta-analysis.