Robustness of Bayesian null hypothesis testing under optional stopping

Abstract

A lot has already been said against the use of optional stopping (i.e., sequential testing) in the frequentist framework. On the other hand, optional stopping under the Bayesian framework is still under contention. In particular, are Bayes factors sensitive to stopping rules? Some researchers have recently argued to this effect (e.g., de Heide & Grünwald, 2018; Sanborn and Hills, 2013; Yu, Sprenger, Thomas, and Dougherty; 2013). However, Rouder (2014) presented arguments defending that Bayes factors are well calibrated under optional stopping (see also Deng et al., 2016). In this talk we will first present evidence that better frames to what extent optional stopping is a problem in frequentist statistics. Afterwards, we offer mathematical arguments in favor of the claims made in Rouder (2014), which were fully simulation-based, for two types of tests. Our mathematical approach is novel, accessible (based only on basic probability theory), and allows clarifying the issue for the tests considered in our study.