Life After P-Hacking

P-hacking is the practice of conducting many analyses on the same dataset seeking to obtain a reportable, statistically significant, result (p < .05). It is likely the major reason why, in many fields (e.g., psychology, medicine), too many published findings do not replicate. Many solutions to the problem of p-hacking have been proposed: abandoning p-values, meta-analytical thinking, Bayesian statistics, registration, pre-registration, focusing on effect size, lowering alpha, etc.  In this talk, I review the proposed solutions, concluding: (i) only pre-registration is a great solution, (ii) most other solutions don't solve the problem but are generally harmless, and (iii) meta-analytical thinking, specially when applied to multiple studies in the same paper, is potentially disastrous.

Note: a co-author of this project, Joe Simmons, gave a talk in 2013 with the same title. That talk, somewhat oddly, is entirely different from this "life after p-hacking" talk.

The talk is largely based on these two papers:

Nelson, Leif D., Joseph Simmons, and Uri Simonsohn. "Psychology's renaissance." Annual review of psychology 69 (2018): 511-534.

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3064811

Vosgerau, Simonsohn, Simmons, Nelson "Don't Do Internal Meta-Analysis: It Makes False-Positives Easier to Produce and Harder to Correct" (working paper available upon request)