The New England Journal of Statistics in Data Science

The study of the efficiency of nonparametric tests that started in the late 1940s is often regarded as a success story in statistics. Some nonparametric tests, such as Wilcoxon’s signed-rank and rank-sum tests, are highly efficient even when used in the framework of popular parametric models, such as the Gaussian model. Theoretical results mostly concern asymptotic efficiency of those tests, but there is also empirical evidence for their finite-sample efficiency. While some nonparametric tests (such as Wilcoxon’s) became very popular after their high efficiency had been discovered, others (such as Wald and Wolfowitz’s run test) were gradually discarded from the statistical literature after their low efficiency had been demonstrated [16, Introduction].

The usual approach to hypothesis testing is based on critical regions or p-values, but in this paper we replace them with their alternative, e-values (see, e.g., [22, 20, 7]). We show that some of the old results about the efficiency of nonparametric tests carry over to hypothesis testing based on e-values. To distinguish our notions of power, tests, etc., from the standard notions, we add the prefix “e-”. (The prefix “p-” is sometimes added to signify standard notions based on p-values, but in this paper we rarely need it since the key notion that we are interested in, Pitman’s asymptotic relative efficiency, is defined in terms of critical regions rather than p-values.)

We explain basics of e-testing in Sect. 2, and in particular, we state an analogue of the Neyman–Pearson lemma in e-testing. In the following section, Sect. 3, we give a simple example of a parametric e-test, one for testing the null hypothesis $N(0,1)$ against an alternative $N(\theta ,1)$ in an IID situation.

In Sect. 4 we give the first, and in some sense most powerful, of the three examples of nonparametric e-tests that we discuss in this paper. It was introduced by Fisher in his 1935 book [5]. Our nonparametric null hypothesis is that of symmetry around 0 (and for simplicity we consider independent observations coming from a continuous distribution).

The material of Sects. 2–4 is standard. After that (Sect. 5) we define the asymptotic relative efficiency of e-tests in the spirit of Pitman’s definition [17]. We regard our definition of asymptotic relative efficiency as a direct translation of the classical definition. Then in Sect. 6 we compute the Pitman-type asymptotic relative efficiency of the Fisher-type test discussed in Sect. 4. This is complemented by similar computations for e-versions of the sign test in Sect. 7 and Wilcoxon’s signed-rank test in Sect. 8. Our results for all three tests agree perfectly with the classical results. This is just a first step, and in Sect. 9 we discuss limitations of our approach (which are considerable) and list natural directions of further research.

Let P be a given probability measure on a sample space Ω (a measurable space). Our null hypothesis is $\{P\}$; it is simple in the sense of containing a single probability measure.

We observe $\omega \in \Omega $ and are interested in whether ω was generated from P. An e-variable for testing P is an $[0,\infty ]$-valued random variable E such that $\textstyle\int E\hspace{0.1667em}\mathrm{d}P\le 1$. In order to be used for testing, we need to choose E before we observe ω. By Markov’s inequality, E can be large only with a small probability (for any threshold $c\gt 1$, $P(E\ge c)\le 1/c$); therefore, observing a large E casts doubt on ω being generated from P.

In the classical Neyman–Pearson approach to hypothesis testing, in addition to P we also have an alternative hypothesis Q. The e-power of an e-variable E is then defined as $\textstyle\int \log E\hspace{0.1667em}\mathrm{d}Q$. This is an analogue of the usual notion of power, but it only works in regular cases. One of such regular cases will be discussed in the next section. The following lemma is very well known (see, e.g., [20, Sect. 2.2.1] and the references therein), and we provide a simple proof.

The likelihood ratio $\mathrm{d}Q/\mathrm{d}P$ in Lemma 1 is understood to be the Radon–Nikodym derivative of Q w.r. to P.

According to Lemma 1, which is an analogue for e-values of the Neyman–Pearson lemma, the optimal e-variable for testing a null hypothesis P against an alternative $Q\ll P$ is the likelihood ratio $\mathrm{d}Q/\mathrm{d}P$. The maximum e-power is (cf. [20, Sect. 2.3] and [7, Theorem 1]). This is simply the Kullback–Leibler divergence [12] of the alternative Q from the null hypothesis P; we will call it the optimal e-power.

We will sometimes refer to $\log E$ as the observed e-power of E; the e-power is then the expectation of the observed e-power w.r. to the alternative hypothesis Q.

The notion of e-power is very close to Shafer’s [20] implied target, the main difference being that the implied target only depends on the null hypothesis P and the e-variable E.

As a short detour, let us check that our notion of e-power enjoys a natural property in testing with multiple e-values. Denote by ${\Pi ^{Q}}$ the function that maps an e-variable to its e-power. Independent e-variables ${E_{1}},\dots ,{E_{K}}$ can be combined into one e-variable using a merging function, the most common choices being convex mixtures of the product functions where M is a subset of $\{1,\dots ,K\}$, with ${F_{\varnothing }}$ set to 1. Denote by $\mathcal{M}$ the convex hull of all functions ${F_{M}}$. Useful elements of the class $\mathcal{M}$ are U-statistics with product as kernel, symmetric merging functions discussed in [22, Sect. 4].

Proposition 1 shows that e-power remains positive when combining independent e-values with positive e-power using a large class of merging functions. As a special case of Proposition 1 applied to only one e-variable, if ${\Pi ^{Q}}(E)\gt 0$, then ${\Pi ^{Q}}(1-\lambda +\lambda E)\gt 0$ for all $\lambda \in (0,1]$. The operation of changing E to $1-\lambda +\lambda E$ is common in building e-processes; see, e.g., [24].

We start our discussion of specific e-tests from a very simple parametric case, that of the Gaussian statistical model ${Q_{\theta }}:=N(\theta ,1)$, $\theta \in \mathbb{R}$, with the variance known to be 1. We observe realizations of independent ${Z_{1}},\dots ,{Z_{n}}\sim N(\theta ,1)$. The null hypothesis P is $N(0,1)$, and we are interested in the alternatives $Q={Q_{\theta }}=N(\theta ,1)$ for $\theta \ne 0$.

For observations ${z_{1}},\dots ,{z_{n}}$ and a given alternative $N(\theta ,1)$, the likelihood ratio of the alternative to the null hypothesis is The corresponding optimal e-power is

The interpretation of the optimal e-power (3.2) usually depends on the law of large numbers and its refinements (such as the central limit theorem and large deviation inequalities). The presence of log in the definition $\textstyle\int \log E\hspace{0.1667em}\mathrm{d}Q$ of the e-power of E under the alternative Q reflects the fact that a typical e-value is obtained by multiplying components coming from the individual observations ${z_{i}}$. This can be seen from (3.1) (and also expressions (4.4), (7.2), and (8.3) below, which are typical). Taking the logarithm leads to a much more regular distribution, which is, e.g., approximately Gaussian under standard regularity conditions. In the case of (3.1), the key component of the logarithm is ${\textstyle\sum _{i=1}^{n}}{z_{i}}$, and we can apply, e.g., the central limit theorem to see that the observed e-power is between the narrow limits $\frac{1}{2}n{\theta ^{2}}\pm c\sqrt{n}\theta $ with probability close (in this particular case, even exactly equal) to $\Phi (c)-\Phi (-c)$, where $c\gt 0$ and Φ is the standard Gaussian cumulative distribution function.

We regard the family (3.1) of e-variables as a test (an e-test) of the null hypothesis $N(0,1)$. While for several important statistical models there are uniformly most powerful p-tests (see, e.g., [14, Chap. 3]), this is not the case for e-tests, and the e-tests considered in this paper are always families of e-variables.

The fact that the e-variable (3.1) depends on the unknown alternative parameter θ is a disadvantage. A natural way out is to integrate it under the prior distribution $N(0,1)$ over θ, which gives us the e-variable (cf. Remark 2 below). Notice that the operation of integration makes the e-variable “two-sided”: while (3.1) is monotone in ${\textstyle\sum _{i}}{z_{i}}$, (3.3) is monotone in $|{\textstyle\sum _{i}}{z_{i}}|$. The remaining disadvantage of the e-variable (3.3) is that it is valid only under the simple Gaussian null hypothesis $N(0,1)$. In the following sections we will replace this simple null hypothesis with a composite nonparametric one.

Let ${Z_{1}},\dots ,{Z_{n}}$ be continuous IID random variables. We are interested in the null hypothesis that their distribution is symmetric around 0. This is an example of a nonparametric hypothesis, since the distribution of ${Z_{1}},\dots ,{Z_{n}}$ is not described in a natural way by finitely many real-valued parameters. Intuitively, we are interested in two alternatives: the one-sided alternative that ${Z_{i}}$, even though IID, are not symmetric but shifted to the right; and the two-sided alternative that ${Z_{i}}$ are shifted to the right or to the left.

A typical case in applications is where ${Z_{i}}:={Y_{i}}-{X_{i}}$, ${X_{i}}$ is a pre-treatment measurement, and ${Y_{i}}$ is a post-treatment measurement, and we are interested in whether the treatment has any effect. Assuming that raising ${X_{i}}$ is desirable, the one-sided alternative is that the treatment is beneficial.

We will formalize our null hypothesis in a way similar to repetitive and one-off structures [23, Sects. 11.2.4 and 11.2.5]. However, we will not need general definitions and will adapt them to our special case.

The symmetry model for a sample size n is the pair $(t,b)$, where $t:{\mathbb{R}^{n}}\to \Sigma $ is the mapping from the sample space ${\mathbb{R}^{n}}$ to the summary space ${[0,\infty )^{n}}$, and b is the Markov kernel that maps each summary ${({z_{1}},\dots ,{z_{n}})\in [0,\infty )^{n}}$ to the uniform probability measure on the set An e-variable for testing the null hypothesis of symmetry is a function $E:{\mathbb{R}^{n}}\to [0,\infty ]$ such that $\textstyle\int E\hspace{0.1667em}\mathrm{d}b(t({z_{1}},\dots ,{z_{n}}))\le 1$ for all ${z_{1}},\dots ,{z_{n}}$. It is admissible if ≤ holds as = for all ${z_{1}},\dots ,{z_{n}}$; in other words, if it ceases to be an e-variable (w.r. to the symmetry model) as soon as its value is increased at any point.

In this section we define the first of our three e-tests for testing symmetry. We are interested in the e-variables of the form where $S({z_{1}},\dots ,{z_{n}}):={\textstyle\sum _{i=1}^{n}}{z_{i}}$, $\lambda \gt 0$ is a positive parameter, and C is chosen to make E an admissible e-variable, i.e., (in other words, $C:=\log \mathbb{E}\exp (\lambda S)$, the expectation being under the null hypothesis, i.e., under the symmetry model). Lemma 2 will give a convenient formula for computing C.

The form (4.2) for our e-variables can be justified by the analogy with the e-variable (3.1) that we obtained in the Gaussian case. The expression for the normalizing constant C will, however, be different and will be derived momentarily.

The justification of the symmetry model from the point of view of standard statistical modelling is that, under the null hypothesis of symmetry, t is a sufficient statistic giving rise to b as conditional distribution.

For simplicity, we will assume that ${z_{1}},\dots ,{z_{n}}$ are all different (under our assumption that the random variables ${Z_{1}},\dots ,{Z_{n}}$ are continuous, the realizations will be all different almost surely).

Plugging (4.3) into (4.2) gives the e-variable This is an e-version of Fisher’s permutation test, which he introduced and applied to Charles Darwin’s data [3, Chap. 1] in his 1935 book [5, Sects. 21 and 21.1] on experimental design.

Again, since there is no uniformly most powerful e-test, we consider a family of e-variables. The e-variable (4.4) is, of course, admissible.

The e-variable (4.4) dominates in the sense ${E^{\prime }}\le E$. Therefore, ${E^{\prime }}$ is also an e-variable, albeit inadmissible in general. To check the inequality ${E^{\prime }}\le E$, it suffices to check that Expanding both sides into Taylor’s series shows that this inequality indeed holds for all x. The inequality is not excessively loose, especially for small values of x (which will be the case that we will be interested in when computing the Pitman efficiencies): cf. Figure 1.

In order to get rid of the dependence of (4.4) or (4.5) on λ, we can integrate these expression over a prior distribution on λ. This can be easily done explicitly (see Remark 2) in the case of (4.5) and the prior distribution $N(0,1)$ on λ:

The right-hand side of (4.7) is close to the right-hand side of (3.3) under $N(0,1)$ as the null hypothesis: this follows from ${\textstyle\sum _{i=1}^{n}}{z_{i}^{2}}\approx n$ (for large n and with high probability). However (as noticed in [4]), this relatively small change drastically changes the property of validity of the e-test: while the right-hand side of (3.3) is an e-test of $N(0,1)$ only, the right-hand side of (4.7) is an e-test of the nonparametric hypothesis of symmetry.

The following definition is in the spirit of Pitman’s definition, which can be found in, e.g., [21, Sect. 14.3]. Let $({Q_{\theta }}\mid \theta \in \Theta )$ be a statistical model, i.e., a set of probability measures on the real line $\mathbb{R}$, with the observations generated from one of those probability measures in the IID fashion. We assume, for simplicity, that $\Theta =\mathbb{R}$ and regard ${Q_{0}}$ as the null hypothesis; informally, the alternative is either one-sided, $\theta \gt 0$, or two-sided, $\theta \ne 0$ (for specific e-tests, we will have the same results for one-sided and two-sided Pitman efficiency). By an e-variable we mean an e-variable w.r. to ${Q_{0}^{n}}$. In our asymptotic framework we consider sequences of parameter values ${\theta _{\nu }}$ that depend on the “difficulty” $\nu =1,2,\dots \hspace{0.1667em}$ of our testing problem; in the one-sided case we will assume ${\theta _{\nu }}\downarrow 0$ (the sequence is strictly decreasing and converges to 0), and in the two-sided case we will assume ${\theta _{\nu }}\to 0$.

Let ${\mathcal{E}_{1}^{n}}$ and ${\mathcal{E}_{2}^{n}}$ be families of e-variables on ${\mathbb{R}^{n}}$; we are interested in the case where ${\mathcal{E}_{1}^{n}}$ is a family of interest to us (a nonparametric e-test such as (4.4) above, or (7.3) or (8.1) below) and ${\mathcal{E}_{2}^{n}}$ is the baseline family of all e-variables on ${\mathbb{R}^{n}}$. The asymptotic relative efficiency of ${\mathcal{E}_{1}^{n}}$ w.r. to ${\mathcal{E}_{2}^{n}}$ is c if, for any $\beta \gt 0$ and any ${\theta _{\nu }}\downarrow 0$ (one-sided case) or ${\theta _{\nu }}\to 0$ (two-sided case), we have ${n_{\nu ,2}}/{n_{\nu ,1}}\to c$, where ${n_{\nu ,j}}$, $j=1,2$, is the minimal number of observations n such that For example, if the asymptotic relative efficiency is 0.5, the best e-test in $({\mathcal{E}_{1}^{n}})$ requires twice as many observations n as the best test in $({\mathcal{E}_{2}^{n}})$ to achieve the same e-power (if the best e-tests exist).

The idea of using an auxiliary parametric statistical model $({Q_{\theta }})$, such as the Gaussian model, to assay the efficiency of nonparametric e-tests is illustrated in Figure 3. We are testing a nonparametric null hypothesis (the hypothesis of symmetry in this paper), but we are afraid that for a popular parametric model (the Gaussian model ${Q_{\theta }}:=N(\theta ,1)$ in this paper, which plays the role of an assay statistical model) our testing method loses a lot. We are interested in the case where the intersection between the nonparametric null hypothesis and the assay model contains only one probability measure; we refer to this intersection as the parametric null hypothesis in Figure 3 (in this paper, it is $\{N(0,1)\}$). For a given simple alternative hypothesis $Q={Q_{\theta }}$ in the assay model (shown as the red dot in Figure 3), we are hoping to show that the best e-power achieved for testing the simple parametric null hypothesis vs Q is not much better than the best e-power achieved for testing the composite (and usually massive) nonparametric null hypothesis. Or, if Pitman-type notion of efficiency is to be used (as in this paper), that the same e-power is attained for numbers of observations that are not wildly different.

Our use of the Gaussian model with variance 1 as assay model motivates using (4.2) with $S({z_{1}},\dots ,{z_{n}}):={z_{1}}+\cdots +{z_{n}}$ as a nonparametric e-test. The sign and Wilcoxon versions will be natural modifications (corresponding to relaxing the symmetry assumption, as explained in Remark 6 below).

For all three nonparametric e-tests considered in this paper (Sects. 6–8 below) we will need the number ${n_{\nu ,2}}$ of observations required by our baseline, which is, by Lemma 1, the likelihood ratio $\mathrm{d}N({\theta _{\nu }},1)/\mathrm{d}N(0,1)$. By (3.2), achieving an e-power of β requires approximately observations (namely, $\lceil 2\beta {\theta _{\nu }^{-2}}\rceil $ observations).

In the classical case, the relative efficiency of Fisher’s test is 1 [6, Chapter 7, Example 4.1], as first shown by Hoeffding [9] (according to Mood [15]). Let us check that this remains true for the e-version as well.

First we find informally a suitable e-variable in the family (4.4) and then show that it requires the optimal number (5.1) of observations to achieve an e-power of β. Under the symmetry model, each observation ${z_{i}}$ is split into its magnitude ${m_{i}}:=|{z_{i}}|$ and sign ${s_{i}}:=\operatorname{sign}({z_{i}})$. Given the magnitudes, the signs are independent and $\mathbb{P}({s_{i}}=1)=1/2$ under the null hypothesis $N(0,1)$ and under the alternative hypothesis $N({\theta _{\nu }},1)$. The conditional likelihood ratio for the signs is This is Fisher’s e-test (4.4) corresponding to $\lambda :={\theta _{\nu }}$. Its observed e-power is Since, under the alternative hypothesis $N({\theta _{\nu }},1)$, and the e-power is We obtain the optimal e-power (3.2) with $\theta ={\theta _{\nu }}$, and so the asymptotic relative efficiency of Fisher’s e-test is 1.

In this and following sections we use (4.2) for different statistics S, and with C still chosen to make ${E_{\lambda }}$ an admissible e-variable. In this section we make the simplest choice of $S({z_{1}},\dots ,{z_{n}})$ in (4.2), which is the number k of positive ${z_{i}}$ among ${z_{1}},\dots ,{z_{n}}$. This gives the sign e-test with parameter $\lambda \gt 0$. The use of the signs for hypothesis testing goes back to [1].

To obtain a useful alternative representation of the sign e-test, let $p\in (0,1)$ be defined by the equation (so that λ becomes the log-odds ratio). The e-variable (4.2) then becomes The last expression is the likelihood ratio of an alternative to the null hypothesis, and so is an admissible e-variable. This gives us the representation of the sign e-test.

The equality between the last two terms in (7.1) gives an explicit expression for C, which in turn gives the alternative representation of the sign e-test.

In view of our informal alternative hypothesis, we are often interested in $\lambda \gt 0$, i.e., $p\gt 1/2$.

As before, we have a dependence of the sign e-test (7.2) on a parameter, p. To get rid of this dependence, we can, e.g., integrate (7.2) over $p\in [0,1]$, obtaining where B is the beta function. For testing the one-sided hypothesis we can integrate (7.2) over the uniform probability measure on $[0.5,1]$, which gives where the second entry of B stands for the incomplete beta function.

Wilcoxon’s signed-rank test [25] is based on arranging the magnitudes $|{z_{i}}|$ of the observations in the ascending order and assigning to each its rank, which is a number in the range $\{1,\dots ,n\}$: the observation ${z_{i}}$ with the smallest $|{z_{i}}|$ gets rank 1, the one with the second smallest $|{z_{i}}|$ gets rank 2, etc. Notice that the symmetry model (i.e., the uniform probability measure on (4.1)) implies that for any set $A\subseteq \{1,\dots ,n\}$, the probability is ${2^{-n}}$ that the observations with the ranks in A will be positive and all other observations will be negative. This determines the distribution (conditional on the magnitudes $|{z_{i}}|$) of Wilcoxon’s statistic ${V_{n}}$ defined as the sum of the ranks of the positive observations.

We will be interested in the nonparametric e-test (4.2) with $S:={V_{n}}$, i.e., The following lemma gives a convenient formula for computing C.

Plugging (8.2) into (8.1), we obtain Wilcoxon’s signed-rank e-test

In the previous sections we mentioned several limitations of our definitions. In this concluding section we will add further details.