Is N-Hacking Ever OK? A Simulation-based study

Abstract

After an experiment has been completed, a trend may be observed that is "not quite significant". Sometimes in this situation, researchers collect more data in an effort to achieve statistical significance. Such "N-hacking" is condemned because it can lead to an excess of false positive results. I use simulations to demonstrate how N-hacking causes false positives. However, in a parameter regime relevant for many experiments, the increase in false positives is quite modest. Moreover, results obtained this way have higher Positive Predictive Value than non-incremented experiments of the same sample size and statistical power. In other words, adding a few more observations to shore up a nearly-significant result can increase the reproducibility of results, counter to some current rhetoric. Many experiments are non-confirmatory, and unplanned sample augmentation with reasonable decision rules would not cause rampant irreproducibility in that context.