The New England Journal of Statistics in Data Science

Consider a sequence of real problems ${E_{1}},{E_{2}},\dots \hspace{0.1667em}$, where ${E_{i}}$ is an experiment yielding data ${x_{i}}$ that arises probabilistically from a distribution having unknown parameter ${\theta _{i}}$, both of which can vary from experiment to experiment; we are not (here) considering the usual frequentist notion of studying repetitions of a fixed experiment with a given distribution and a fixed unknown θ.

The scenario considered, in this section, is that of producing confidence intervals for the ${\theta _{i}}$, so the result of each analysis is a confidence interval ${C_{i}}({x_{i}})$, with stated confidence (of containing ${\theta _{i}}$) equal to $1-{\alpha _{i}}({x_{i}})$. We are not defining ‘confidence’ here – it could be either frequentist or Bayesian, for instance – and we allow the stated confidence to depend on the data.

Suppose one eventually learns if ${C_{i}}({x_{i}})$ contains ${\theta _{i}}$ or not (say we are targeting stock prices in the future, and eventually learn them). The empirical frequentist principle could be formulated, in this context, as saying that Thus the difference between the average reported confidence and the average attained coverage of the intervals should go to zero. It is often viewed as being acceptable to be conservative, which would happen, for instance, if the bracketed term in (2.1) were less than zero for sufficiently large N.

What is random in (2.1) will depend on context; typically $({x_{1}},{x_{2}},\dots ,{x_{N}})$ will be random, with a joint distribution specified by the distributions in $\{{E_{1}},\dots ,{E_{N}}\}$, given $({\theta _{1}},\dots ,{\theta _{N}})$. But sometimes the ${\theta _{i}}$ will also be random. (And sometimes neither the ${x_{i}}$ nor ${\theta _{i}}$ are random; in finite population settings, for instance, both are considered fixed and the randomness comes from the random mechanism by which subjects are selected to be in the sample.) Such considerations arise when trying to prove that (2.1) holds, but the condition itself does not depend on any notion of randomness.

The textbook notion of confidence is, however, one possible report and can satisfy the empirical frequentist principle. Indeed, suppose that, in ${E_{i}}$, for all ${\theta _{i}}$ (the probability is over the possible data ${x_{i}}$), i.e., $1-{\alpha _{i}}$ is the usual frequentist coverage of the confidence procedure in ${E_{i}}$. We will evaluate (2.1) when reporting $1-{\alpha _{i}}({x_{i}})=1-{\alpha _{i}}$ as the error.

Letting ${1_{C}}(\theta )$ be the indicator function on the set C (one if $\theta \in C$ and zero otherwise), note that (2.1) can be rewritten Also, $1-{\alpha _{i}}=E[{1_{{C_{i}}({x_{i}})}}({\theta _{i}})\mid {\theta _{i}}]$ (the expectation being over ${x_{i}}$), so ${y_{i}}=[1-{\alpha _{i}}-{1_{{C_{i}}({x_{i}})}}({\theta _{i}})]$ is a zero mean random variable with variance bounded by 1. Since (2.2) is just the average of the ${y_{i}}$, the law of large numbers applies and concludes that the limit is indeed zero. Thus the ${E_{i}}$ do not have to be problems of the same type, the ${\alpha _{i}}$ do not have to be the same, and the ${\theta _{i}}$ need have no relationships for (2.1) to hold.

Consider a sequence of different experiments ${E_{i}}$, with different ${\theta _{i}}$, to be estimated by unbiased estimates ${\hat{\theta }_{i}}$ (so that $E[{\hat{\theta }_{i}}\mid {\theta _{i}}]={\theta _{i}}$). If the ${\theta _{i}}$ were to become known, the differences ${\hat{\theta }_{i}}-{\theta _{i}}$ would then be mean 0 random variables and one could observe that the average differences converge to 0, under mild conditions. Whether or not this is a useful property can be debated, but it is an empirical frequentist property.

Empirical Bayes analysis [21] is defined in an empirical frequentist way. The ${\theta _{i}}$ are assumed to arise from some unknown distribution $\pi (\cdot )$; for instance they could be independent draws from a $N({\theta _{i}}\mid \xi ,{\tau ^{2}})$ distribution, with ξ and ${\tau ^{2}}$ unknown. Data ${x_{i}}$ are then observed from distributions with parameters ${\theta _{i}}$. Analysis is then typically done with respect to the series of experiments corresponding to the $({x_{i}},{\theta _{i}})$, and often in accordance with the empirical frequentist principle, under the assumption that the ${\theta _{i}}$ do arise from $\pi (\cdot )$.

The empirical frequentist principle seems compelling to most people. Imagine that a computer program to compute confidence intervals has been developed. Many different people use the program, specifying the $1-{\alpha _{i}}$ they want and the experiment ${E_{i}}$ they conducted, and the program returns ${C_{i}}({x_{i}})$. Suppose someone discovers that, over many uses, the average reported confidence was 0.90, while only 70% of the confidence intervals actually contained the true ${\theta _{i}}$. This would be a very misleading computer program.

We would argue that even Bayesians should accept the empirical frequentist principle. Perhaps the computer program above is a subjective Baysian program that guides users through the prior elicitation process and ultimately produces Bayesian confidence (credible) intervals. Something is very wrong if the reported confidence in repeated use of the program is 0.90, with actual coverage of only 70%. This could arise, for instance, if the subjective prior elicitation part of the program is producing prior distributions that are too concentrated (a well known issue with subjective elicitation, unless considerable care is taken), leading to credible intervals that are too small.

Confidence is often defined in a procedural frequentist way, with the experiment E being fixed, the unknown θ being fixed, and confidence $1-\alpha (\theta )$ being defined as the probability that the confidence set contains θ for imaginary repetitions of the experiment.(We are overusing α in this paper; $\alpha (\theta )$ here is distinct from the earlier $\alpha ({x_{i}})$.) As observed earlier, if one develops confidence sets in the procedural frequentist way and the confidence $1-\alpha $ does not depend on θ, the confidence procedures will also have the empirical frequentist property. When the confidence does depend on θ, it is not uncommon to report $1-\alpha ={\inf _{\theta }}(1-\alpha (\theta ))$ and such reports can be given a conservative empirical frequentist interpretation, with ≤ replacing = in (2.1).

A procedure is consistent if it converges to the truth as the sample size $n\to \infty $. (Note that this is distinct from the earlier N, which referred to the sequence of experiments being conducted.) This is a procedural frequentist principle, in that it involves an imaginary sequence of applications of the procedure to a given problem (model), but with growing sample size. There is no natural sense in which this is an empirical frequentist principle; one does not, in reality, continue to repeat the same experiment, but with growing sample sizes.

Consider testing ${H_{0}}$ versus ${H_{1}}$, with a rejection region $\mathcal{R}$ having Type I error $\alpha =P(\mathcal{R}\mid {H_{0}})$. This is clearly a procedural frequentist quantity but we will see in Section 3.2 that it does not satisfy the empirical frequentist principle.

Consider a sequence of null and alternative hypotheses $\{{H_{0}^{1}},{H_{1}^{1}}\}$, $\{{H_{0}^{2}},{H_{1}^{2}}\}$, …, that are to be tested sequentially; the ordering of the hypotheses is important, and must be pre-specified. For instance ${H_{1}^{1}}$ could be the hypothesis that a new drug provides pain relief, ${H_{1}^{2}}$ could be the hypothesis that the same drug reduces blood pressure, and ${H_{1}^{3}}$ could be the hypothesis that the same drug promotes weight loss. Indeed, this type of example motivated the name sequential endpoint testing, with the three possible effects of the drug being the particular endpoints being studied.

In the simplest version of sequential endpoint testing, the same Type I error, α, is chosen for each hypothesis test. The procedure is to conduct the first test, stopping if ${H_{0}^{1}}$ is not rejected. If ${H_{0}^{1}}$ is rejected, one is allowed to perform the second test, stopping or continuing on depending on whether the second test fails to reject or rejects. Continuing on in this fashion, the end result is some sequence (possibly empty) $\{{H_{0}^{1}},{H_{0}^{2}},\dots ,{H_{0}^{m}}\}$ of rejected null hypotheses, with $m+1$ being the first time one fails to reject. The interesting procedural frequentist fact [17] is that no matter what sequence of hypotheses is true. So, in the drug illustration and if all three tests are rejections, the drug company could claim that the drug is effective for all three purposes, with the probability that the procedure results in one or more incorrect rejections being no more than α. This is what now occurs in the world, with a drug often being labeled as effective for several things, based on sequential endpoint testing.

This at first seems odd to statisticians because it looks similar to traditional multiple testing for which, to obtain an overall level of α for the three tests, one would need to do the individual tests at level $\alpha /3$ (using Bonferonni, for simplicity). In sequential endpoint testing, however, not all tests are necessarily conducted; a test is conducted only if all the preceding tests were rejections, the crucial reason that no Type I error correction is needed. We show, however, in Section 3.2.3 that this procedure does not satisfy the empirical frequentist principle.

The procedural frequentist principle is less compelling than the empirical frequentist principle, in that it involves an imaginary sequence of experiments. The consistency example is one in which many (most) people would find the principle compelling, even though it is only procedural; using a procedure that fails as the data becomes nearly infinite provides a thought experiment that calls the procedure into question. Even Bayesians routinely accept consistency as necessary.

The procedural case for the Type I error, α, is not so compelling: one considers an imaginary sequence of experiments consisting of draws of data from ${H_{0}}$, and notes that the proportion of the time that the data is in $\mathcal{R}$ is α. Since this sequence of experiments is all under the assumption that ${H_{0}}$ is true, it is not obvious that one has learned much about the testing problem; this will be extensively discussed in Section 3.2. Of course, Type I error is a useful quantity for various other computations. In particular, when designing an experiment, Type I error and power are key quantities to consider, even for a Bayesian. (But, once the data are at hand, a Bayesian would not tend to utilize Type I error or power in making an error report.)

Procedural frequentist properties are often used by objective Bayesians to define objective priors. For instance, the confidence set procedural principle is used to define what are called matching priors, which are priors that yield posterior credible sets for a real parameter θ that have good frequentist behavior when viewed as a confidence procedure.

The procedural frequentist procedure defined in sequential endpoint testing is incompatible with Bayesian reasoning. In particular, the Bayesian posterior probabilities of the alternative hypotheses satisfy $P({H_{1}^{1}}\mid data)\ge P({H_{1}^{1}},{H_{1}^{2}}\mid data)\ge \cdots \ge P({H_{1}^{1}},{H_{1}^{2}},\dots {H_{1}^{m}}\mid data)\hspace{0.1667em}$, so that increasing numbers of rejections result in less probability being assigned to all the rejections being correct. Indeed, this seems intuitively clear. If one managed to conduct 101 $\alpha =0.05$-level tests via the sequential endpoint procedure, with each of the endpoints being very different (as in the earlier drug illustration), would anyone actually be willing to bet that all 100 rejections were correct? That seems scientifically ridiculous. Of course, this outcome has a nearly negligible probability of occurring and, hence, can be compatible with an overall $\alpha =0.05$. But the possibility of being in this situation sends a warning that the procedural frequentist property here is difficult to interpret. Indeed, even assigning the same error probability to one rejection as to two or three rejections seems highly questionable.

We discussed four procedural frequentist examples in this section. The first, that of textbook presentation of coverage of confidence intervals, actually has an empirical frequentist justification, so there can be no criticism of it at this point (although see Section 2.4). The second example, that of consistency, has no clear empirical frequentist justification, but is almost universally agreed to be important.

The third example, that of Type I error in testing of a single hypothesis, has no clear empirical frequentist justification and is not as universally respected as consistency, but can be an important quantity to know. But the final example, that of sequential endpoint testing, is a highly suspect procedural frequentist procedure.

In testing ${H_{0}}$, based on data x, where large values of $T(x)$ discredit ${H_{0}}$, the p-value $P(T(x)\ge T({x_{obs}})\mid {H_{0}})$, where ${x_{obs}}$ is the actual observation, is a probability on the sample space, so it satisfies the computational frequentist principle. Note, however, that it does not follow the procedural frequentist principle, because one cannot embed it in an imaginary sequence of problems where the p-value has a long-run frequentist interpretation. For instance, one might consider the imaginary experiments of repeatedly drawing data ${x_{j}}$ from ${H_{0}}$, computing the p-value $p({x_{j}})$, rejecting ${H_{0}}$ if $p({x_{j}})\lt 0.05$ and then reporting $p({x_{j}})$ as a frequentist error probability. But the actual Type I frequentist error of this procedure is clearly 0.05, so that reporting p-values will always underestimate the procedural error. We will also see that the p-value badly fails to satisfy the empirical frequentist principle.

The p-value is often called the ‘attained significance level,’ in that it is the smallest α for which an α-level test would have rejected. One could imagine then running a sequence of imaginary experiments with this α, but this imaginary sequence will have a long-run frequentist interpretation only if α is used as the error probability, not if p-values are used in the new imaginary experiments.

Note that computationally frequentist arguments can be ridiculous. Here is an example, arising from an e-mail we received that was inquiring about the validity of the analysis.

Our perspective is that the computationally frequentist principle lacks any force whatsoever. Just because one has computed some kind of average over the data does not mean that the resulting procedure has any value. We are not asserting that any procedure arising this way is useless, but merely saying that the fact that there was some data averaging going on provides no justification by itself.

p-values are interesting in this regard; they are valuable statistics and have a number of important uses, especially if they are properly calibrated; even Bayesians tend to use (calibrated) p-values for model checking. But Bayesians do not view their value as arising from the fact that they involve an average over data but, rather, that they are just a useful statistic.

Computationally frequentist procedures will generally not have a Bayesian interpretation, although sometimes they do through a mathematical quirk. For instance, in one-sided testing, p-values can equal the posterior probability of the null hypotheses for certain improper priors, but this is more a mathematical curiosity than something fundamental (cf, [4]). In two-sided testing, there is usually an extreme difference between p-values and posterior probabilities, a fact first clearly demonstrated in [10].

Finding good conditioning statistics is, in general, very difficult – so much so that the conditional frequentist theory of statistics is quite underdeveloped. Thus the typical approach today for developing conditional frequentist procedures is to develop objective Bayesian procedures (which automatically condition correctly) and show that they have excellent long-run frequentist behavior. The generalized fiducial approach mentioned in the introduction is another promising approach for doing this.

To illustrate this on the two-observation example in the previous section, the natural objective prior is $\pi (\theta )=1$. Application of Bayes theorem trivially yields that, if ${x_{1}}\ne {x_{2}}$, then the posterior distribution for the unknown θ gives probability one to the point $({x_{1}}+{x_{2}})/2$ while, if ${x_{1}}={x_{2}}$, then the posterior distribution gives probability 1/2 each to the common value of the data plus 1 and the common value minus 1. It is immediate that the objective Bayesian confidence statements for $C({x_{1}},{x_{2}})$ are 1 and 0.5 for the two cases, respectively, which is the optimal conditional frequentist answer.

The example in this section showed that even satisfaction of the empirical frequentist principle can be highly inadequate from the conditional frequentist perspective. (This could be corrected within the empirical frequentist paradigm by requiring some type of second empirical frequentist property, involving losses such as (2.4), but we do not pursue this.) This will be seen to be even more of a problem for procedures that satisfy only the procedural frequentist principle, as will be extensively discussed in the next section.

Consider the case of exchangeable simple hypothesis testing, with each of the ${E_{i}}$ (recall that $\{{E_{1}},{E_{2}},\dots ,{E_{N}}\}$ is the sequence of experiments being considered) being a test of ${H_{0}^{i}}:{\theta _{i}}={\theta _{0i}}$ versus ${H_{1}^{i}}:{\theta _{i}}={\theta _{1i}}$, with rejection regions ${\mathcal{R}_{i}}$ having the same Type I error α and the same power $\beta =P({\mathcal{R}_{i}}\mid {H_{1}^{i}})$. There are various empirical frequentist properties that can be discussed in testing. For simplicity, we will focus on what could be called the empirical frequentist error probability under rejection, namely which is the posterior probability that ${H_{0}^{i}}$ is true if one only knows that the test was a rejection. This is the actual error rate achieved in the sequence of experiments and so is the target for our reported error probabilities.

If ${\pi _{0}}$ is known, simply reporting error probability ${\alpha _{U}}=P({H_{0}^{i}}\mid {\mathcal{R}_{i}})$ clearly satisfies the empirical frequentist principle. Furthermore, this would be the correct error report corresponding to the experiment, before seeing the data. In [23], this was shown to be equal to what he defined as the pFDR. Thus stating pFDR (if ${\pi _{0}}$ is known) has an empirical frequentist justification. Note that the expected value of the first bracketed quantity above is essentially the regular FDR [2], which has a procedural frequentist justification but not an empirical frequentist interpretation.

If ${\pi _{0}}$ is not known and one makes the default assumption that ${\pi _{0}}=0.5$, note that $P({H_{0}^{i}}\mid {\mathcal{R}_{i}})=\alpha /[\alpha +\beta ]$, which is nearly α when α is small and β is near one. Thus, for a highly powered test and if the hypotheses have equal prior probabilities, reporting α as the error probability does have approximate empirical frequentist justification.

The dependence of empirical frequentist error on prior probabilities is circumvented in Neyman-Pearson testing by only evaluating procedural properties of the test, namely α and β individually. The problem with this is that it is not unusual for people to interpret α as a surrogate for the empirical frequentist quantity $P({H_{0}^{i}}\mid {\mathcal{R}_{i}})$, but the two can obviously be very different, even if ${\pi _{0}}=1/2$, as shown in Table 1, where $\alpha =0.05$ is chosen to define the rejection region. Thus, if the power is only 0.5 (as is common in GWAS), the actual error that will arise in rejecting over a series of real experiments is almost twice α and, as the power drops lower, the actual error rises dramatically. This important role of power in achieving empirical frequentist performance is often neglected, in part because it is not usually explained how to utilize power to understand empirical frequentist rejection error. We return to this issue in Section 3.4.

Consider the multiple testing scenario in which ${E_{i}}$ consists of performing m independent tests of hypotheses at nominal Type I error $\alpha /m$ (the Bonferroni correction) and power $\beta (m)$. The Type I error (a procedural frequentist quantity) for each ${E_{i}}$ is then α, and we again study the extent to which this report has empirical frequentist justification. One could allow the m to vary over the ${E_{i}}$ and the β’s to vary – between both the ${E_{i}}$ and the tests within each ${E_{i}}$ – but the answers remain essentially the same. Also, assume for simplicity that each null hypothesis has prior probability ${\pi _{0}}\lt 1$ of being true.

Consider the situation in which an error is made in ${E_{i}}$ if any of the m tests results in an incorrect rejection (often called family-wide error in rejection). Then one natural empirical frequentist quantity to study is the false positive rate for family-wide error in rejection. There are other possibilities here, such as looking at all the tests within each ${E_{i}}$ and studying the overall number of tests being incorrectly rejected, but utilizing family-wide error is standard.

For large m, the numerator in (3.3) is approximately $1-{e^{-{\pi _{0}}\alpha }}\approx {\pi _{0}}\alpha $ for small α. Typically, $\beta (m)$ goes to 1 as m grows, in which case an approximation to the denominator (and always a lower bound) can be shown to be $[1-{\pi _{0}^{m}}(1-\alpha )]$. Thus

To study this, it is important to realize that ${\pi _{0}}$ is often near 1 when m is large. For instance, in [24], m was huge and ${\pi _{0}}=1-{10^{-5}}$. It is thus useful to consider three types of behavior of ${\pi _{0}}$, with regards to increasing m.

We return to the sequential endpoint testing example, and evaluate it from the empirical frequentist perspective. To keep matters simple, the only case that will be considered is that in which each endpoint test is conducted with the same Type I error α and power β, and the prior probability of each null hypotheses is ${\pi _{0}}$. Note that ${E_{i}}$ is again a possible sequence of individual tests; thus ${E_{99}}$ could be a sequence in which ${H_{0}^{1}}$ is rejected, ${H_{0}^{2}}$ is rejected, and ${H_{0}^{3}}$ is not rejected. (Recall that the only possible outcomes are of this type: a sequence of rejections followed by an acceptance.)

Of interest is again the actual empirical frequentist rejection error rate among the sequences that contain at least one rejection, namely the quantity (3.2).

Calculus allows computation of the minimum of (3.5) over β, resulting in the inequality This lower bound can be shown to always exceed α, for small α, when ${\pi _{0}}\gt \frac{1}{4}+\frac{\alpha }{8}$, so that anytime the null hypotheses have even modest probability of being true, sequential endpoint testing will not satisfy the empirical frequentist principle when measured by (3.5).

For the objective choice ${\pi _{0}}=1/2$ and α small, the above bound is approximately $2\alpha $ and so stating that the error is α understates the error by a factor of 2. Even reporting $2\alpha $ as the error does not satisfy the empirical frequentist principle because the inequality above is in the anti-conservative direction.

Similar analysis for sequential endpoint testing consisting of just m steps can be performed and yields lower bounds for the empirical frequentist rejection error (when ${\pi _{0}}=1/2$ and α is small) of $2(1-{2^{-m}})\hspace{0.1667em}\alpha $. For instance, if $m=2$, this is $(1.5)\hspace{0.1667em}\alpha $, which is 50% larger than α. The clear indication is that, even though sequential endpoint testing does not get penalized in terms of Type I error for using α as the rejection level for each test, there is a penalty in terms of empirical frequentist rejection error.

Again, we only consider the case of exchangeable simple hypothesis testing, with each of the ${E_{i}}$ being a test of ${H_{0}^{i}}:{\theta _{i}}={\theta _{0i}}$ versus ${H_{1}^{i}}:{\theta _{i}}={\theta _{1i}}$, with rejection regions ${\mathcal{R}_{i}}$ having Type I error α and power $\beta =P({\mathcal{R}_{i}}\mid {H_{1}^{i}})$, and ${\pi _{0}}$ being the prior probability of ${H_{0}^{i}}$. There are various possible choices for data-dependent error probabilities ${\alpha _{i}}$. Instead of working with the data, it is convenient to work with the p-values ${p_{i}}$ (against the null hypotheses), and write ${\alpha _{i}}({p_{i}})$ as the reported error probability upon rejecting in ${E_{i}}$. (The ${p_{i}}$ are only being used as convenient statistics here.) Recall that the target is the actual empirical frequentist error probability $P({H_{0}^{i}}\mid {\mathcal{R}_{i}})$ in (3.1), so the ideal is for the ${\alpha _{i}}({p_{i}})$ to satisfy where ${N^{\ast }}$ is the number of rejections and the average is over the ${\alpha _{i}}({p_{i}})$ in the rejections.

Under the null hypotheses, the ${p_{i}}$ have a uniform density on $(0,1)$ (assuming they are proper p-values). Let ${f_{1}}(p)$ denote the density of the ${p_{i}}$ under the alternative hypotheses, the density being common across the ${E_{i}}$ because of the exchangeability assumption. The following lemma follows directly.

This suggests an obvious data-dependent error probability report when ${\pi _{0}}$ is known, namely For this choice, the right hand side of (3.7) clearly equals $P({H_{0}^{i}}\mid {\mathcal{R}_{i}})$, achieving exact empirical frequentist justification. In addition to this justification, these reported error probabilities have the highly desirable property of being data-dependent, with the reported error probability decreasing as the p-value decreases. As discussed in the conditional frequentist section, this is thus a much better frequentist report than $P({H_{0}^{i}}\mid {\mathcal{R}_{i}})$.

When ${\pi _{0}}$ is known, there is thus no frequentist controversy: report $P({H_{0}^{i}}\mid {\mathcal{R}_{i}})$ as the pre-experimental error probability under rejection but, upon observing the data, report ${\alpha _{B}}({p_{i}})$. Note that ${\alpha _{B}}({p_{i}})=P({H_{0}^{i}}\mid {p_{i}})$, i.e., is the posterior probability of the null hypothesis, given the data. The fact that the Bayesian error probability here is also the optimal empirical frequentist error probability was noted and discussed in [8].

When the alternative hypothesis is not simple (as assumed here), then ${f_{1}}(p)$ will also not be known. A method for dealing with this is discussed in the next section.

When ${\pi _{0}}$ is unknown, the following are commonly considered ‘objective’ conditional error reports.

The empirical frequentist performance of these data-dependent error probabilities will be studied by considering the ratio The (conservative) empirical frequentist principle is satisfied if $R\ge 1$ (the average reported error is not less than the average actual error), which will be true if

From Tables 2 and 3, it is clear that reporting the p-value as the error probability is terrible according to the empirical frequentist principle; it is reasonable only when the (unknown) ${\pi _{0}}$ is very small. And the underreporting of error for the ‘objective’ ${\pi _{0}}=0.5$ is dramatic.

The conditional frequentist (objective Bayesian) report ${\alpha _{C}}({p_{i}})=\frac{1}{1+{f_{1}}({p_{i}})}$ is clearly very reasonable, needing only ${\pi _{0}}\le 0.5$ to have (conservative) empirical frequentist justification. As argued in [1], it is often the case that ${\pi _{0}}\gt 0.5$, but then one should be performing a subjective Bayesian analysis.

Reporting ${\alpha _{O}}({p_{i}})=-e{p_{i}}\log {p_{i}}/[1-e{p_{i}}\log {p_{i}}]$ is clearly considerably better than reporting the p-value in terms of the empirical frequentist principle, becoming much too small only for very small p-values. It can be shown that p is a factor of at least 3.85 smaller than ${\alpha _{O}}(p)$ when $p\lt 0.1$, so reporting p is almost 4 times worse than reporting ${\alpha _{O}}({p_{i}})$. For the case of simple hypothesis testing considered here, one could use the superior ${\alpha _{C}}({p_{i}})=1/[1+{f_{1}}({p_{i}})]$ with no additional computational cost but, for more general hypothesis testing problems, it can be difficult to compute the objective Bayesian error probability, while computing ${\alpha _{O}}({p_{i}})$ is as easy as computing the p-value.

It is surprising that ${\alpha _{O}}({p_{i}})$ has $R\gt 1$ when $n=1$ and $n=2$, implying that the inequality in (3.9) can fail for larger p-values. The inequality was established under a certain condition on the hazard rate corresponding to ${f_{1}}(p)$, and this condition is apparently violated for the simple hypothesis example considered here, when $n=1$ and $n=2$. In more general composite hypothesis testing problems arising in practice, the inequality does seem to hold and, interestingly, ${\alpha _{O}}({p_{i}})$ seems to often be quite close to ${\alpha _{C}}({p_{i}})$, as shown in empirical studies from [16] (see also [1]). Thus, general use of ${\alpha _{O}}({p_{i}})$ seems justified, even if it does not always strictly satisfy the empirical frequentist principle.

One might consider a data dependent version of procedural frequentism. For instance, one could propose evaluating data dependent Type I errors ${\alpha _{i}}({p_{i}})=\alpha ({p_{i}})$ (for some function $\alpha (\cdot )$) by looking at an imaginary sequence of ${N^{\ast \ast }}$ rejections under the null hypothesis, and ask that One might then claim that reporting $\alpha ({p_{i}})$ is as frequentist as reporting α. As an example, choosing $\alpha ({p_{i}})=2{p_{i}}$ yields $\frac{1}{\alpha }{\textstyle\int _{0}^{\alpha }}2pdp=\alpha $, so one might assert that reporting twice the p-value is as much a procedural frequentist procedure as reporting α.

This is not, however, a logical conclusion. α is the procedural frequentist property of the test, and the $\alpha ({p_{i}})$ have no real meaning in terms of procedural frequentism.

It is, however, possible to develop data dependent procedural tests through conditioning. Indeed, [8] considers testing conditional on the statistic $S=\max \{{p_{0}},{p_{1}}\}$, where ${p_{0}}$ is the p-value under ${H_{0}}$ and ${p_{1}}$ is the p-value under ${H_{1}}$. They show that the conditional Type I error, given S, is, for appropriate rejection regions $\mathcal{R}$, given by $\alpha (S)=P(\mathcal{R}\mid {H_{0}},S)=\frac{1}{1+{f_{1}}({p_{0}})}\hspace{0.1667em}$. This is a real procedural frequentist quantity, having the interpretation as the Type I error arising in a long series of experiments under ${H_{0}}$, where the data is compatible with the specified S. Noting that $\alpha (S)$ is always much bigger than $2{p_{i}}$ further reinforces the notion that satisfaction of (3.12) does not provide any procedural frequentist validity.

Recalling that ${\pi _{0}}$ is the prior probability of ${H_{0}^{i}}$, Bayes theorem gives which is commonly rewritten in terms of odds as using the terminology in [1]. The pre-experimental odds have the very nice interpretation as the odds that a rejection, arising from the experiment, is correct to incorrect (often also called the odds of a true positive to a false positive). The big advantage of expressing things in terms of odds is that the prior odds separate out, so that those who do not wish to involve prior probabilities can focus on the experimental odds.

In classical statistics, it is left unstated as to how one should combine α and β to make inferences. Combining them through error probabilities, as in Section 3.2, is one possibility, but this mixes α and β up with the prior probabilities of hypotheses; (3.14) makes the sharper statement that inferences should depend on α and β only through the ratio $\beta /\alpha $.

Consider (possibly data-dependent) reports ${O_{i}}({p_{i}})$ of the odds of having a correct rejection to an incorrect rejection in experiment ${E_{i}}$. A natural empirical frequentist principle would then be to satisfy, averaging over all ${N^{\ast }}$ rejections, With error probabilities it was natural to evaluate their long run performance by arithmetic averaging, but this is not so natural with reported odds. Indeed, using either geometric averaging or arithmetic averaging of log odds in (3.15) may be more reasonable.

Note that, if the prior odds are known, the unconditional choice ${O_{i}}=(1-{\pi _{0}})\beta /({\pi _{0}}\alpha )$ trivially satisfies (3.15), which is strong empirical frequentist justification for the choice. (This would also be trivially true under geometric averaging.) If the prior odds are unknown, one can, at least, provide a procedural frequentist justification for $\beta /\alpha $ by considering an imaginary sequence of tests in which the prior odds are fixed at some specified value ${O^{\ast }}$ (e.g., the objective choice ${O^{\ast }}=1$), and then saying that ${O_{i}}={O^{\ast }}\beta /\alpha $ will satisfy (3.15) for this imaginary sequence.

(3.13) is a version of Bayes theorem applied pre-experimentally, depending only on ${\mathcal{R}_{i}}$. The post-experimental odds version of Bayes theorem is where, for our testing problem, ${B_{10}}({p_{i}})={f_{1}}({p_{i}})/1$ (the density of the statistic ${p_{i}}$ under the alternative hypothesis divided by the density under the null hypothesis).

Turning to the empirical frequentist performance of reporting ${B_{10}}({p_{i}})$, computation yields the last step following from Jensen’s inequality, since

If the prior odds ${O_{i}}$ are known, the conditional report would be ${O_{i}}{B_{10}}({p_{i}})$. Thus (3.17) implies that these reports do not have an empirical frequentist justification (the target being ${O_{i}}\beta /\alpha $). This is still useful as a bound, however: the odds in favor of ${H_{1}^{i}}$ cannot be larger than ${O_{i}}{B_{10}}({p_{i}})$.

Algebra shows that (see (3.8)) $P({H_{0}^{i}}\mid {p_{i}})={\alpha _{B}}({p_{i}})=1/(1+{O_{i}}{B_{10}}({p_{i}}))$, and we saw in Section 3.3.2 that this is the optimal empirical frequentist error probability. Thus, if we had defined empirical frequentist performance of posterior odds by averaging the $1/(1+{O_{i}}{B_{10}}({p_{i}})$), the posterior odds approach would also be optimal. This would be a rather strange way to average odds, however.

One could have, instead, stated the odds of ${H_{0}^{i}}$ to ${H_{1}^{i}}$. The relevant overall frequentist quantity would then have been $\alpha /\beta $, while the Bayes factor would be ${B_{01}}({p_{i}})=1/{f_{1}}({p_{i}})$. Now the empirical frequentist property would be the last step again following from Jensen’s inequality, since So, from an empirical frequentist perspective, one is now overstating the evidence in favor of ${H_{0}^{i}}$, which could be viewed as being conservative.

Finally, we consider an argument given in [1] concerning the data dependent reports ${B_{10}}({p_{i}})$. Averaging these over an imaginary sequence ${N^{\ast \ast }}$ of rejected true hypotheses ${H_{0}^{i}}$ yields Thus it was claimed, in [1], that reporting the ${B_{10}}({p_{i}})$ under the ${H_{0}^{i}}$ has the same long run procedural frequentist justification as reporting $\beta /\alpha $. But $\beta /\alpha $ is the procedural frequentist quantity here and it is not clear that the ${B_{10}}({p_{i}})$ have any such justification (as was the case for the related interpretation of (3.12)). Recall, however, that ${B_{10}}({p_{i}})$ did have partial empirical frequentist justification.

Our conclusion about hypothesis testing is that, if the prior probabilities of the hypotheses are known, estimable or given (as in the objective choice of 1/2 each), then reporting ${\alpha _{B}}({p_{i}})$ is the optimal empirical frequentist error probability (also the optimal Bayesian error probability), because it exactly satisfies the empirical frequentist property, while being fully data-dependent. If prior probabilities are unknown and one is not willing to make the objectivity assumption, the situation is less clear, with the only compelling conclusion being that reporting the p-value as the error probability is terrible from the empirical frequentist perspective.

This lack of clarity, when prior probabilities are unknown, seems to argue for focusing on odds, rather than error probabilities, because one can then clearly separate prior odds and experimental odds. Unfortunately, $\beta /\alpha $ only has a nice empirical frequentist interpretation when the prior odds are known, although it always has a procedural frequentist interpretation. The Bayes factor ${B_{10}}({p_{i}})$ does not exactly satisfy the empirical frequentist principle, even when the prior odds are known. So, based on frequentist reasoning alone, the situation with odds is murky. However, we could have, instead, averaged the $1/(1+{O_{i}}{B_{10}}({p_{i}}))$, and then the posterior odds would have been the optimal empirical frequentist report.

This section only considered testing of simple hypotheses. References where these issues are discussed in more complicated testing scenarios, from a conditional frequentist perspective, include [6, 7], [9], and [1].