Detection coherence of tests

Abstract

There exist tests calibrated under a null narrower than the one implied by their test statistic -- the detection-null set. The part of the detection-null set not in the null is the test's blind spot. A framework for assessing detection coherence -- whether a test's blind spot is empty -- is introduced, built on new concepts of calibration statistic, detector, detection functional, detection-null set, and blind spot. It is demonstrated that the Wilcoxon--Mann--Whitney, Kruskal--Wallis, Friedman, and logrank tests are detection incoherent as unrestricted tests. Detection incoherent tests may become coherent under restrictions that make the blind spot empty, though domain restriction is not a reliable route to coherence in practice. The Kolmogorov--Smirnov and Zaremba tests, and the Maximum Mean Discrepancy test with a characteristic kernel, are shown to be universally detection coherent. Due to the blind spot, a detection incoherent test misses discoveries when non-rejecting and yields spurious discoveries when rejecting, in both cases regardless of sample size. Detection incoherent tests should be abandoned.