The logrank test is arguably the most important tool for the statistical comparison of time-to-event data between two groups of participants. Our main focus is when the two groups refer to the treatment and control groups in a randomized controlled trial; the outcome of interest are event times, that is, the time elapsed until an outcome of interest. The logrank test, in turn, uses a simplified version of the proportional hazard ratio model of Cox [3]. For a fixed sample size and under this model, Cox gave a simple but profound insight: inference can be performed using the partial likelihood of having observed the events in the particular order that they were observed. To this end, the logrank test [20, 24], the score test associated to the Cox’ partial likelihood, is optimal for fixed sample size and a restricted alternative. Large-sample properties of the logrank test are known in very general settings [41, 32, 1]. Nevertheless, it is clear that the fixed-sample regime can be overly restrictive. Indeed, due to ethical and practical constraints in human survival-time medical trials, interim analyses may be performed to terminate the study earlier than planned if needed. Consequently, it has been of fundamental importance to develop methods for the sequential analysis of time-to-event data in general; for the logrank test, in particular.

In order to legitimate the use of sequential boundary decisions, asymptotic approximations over the study period have been developed for the logrank statistic [42, 36, 38]. The results in this line of work show the convergence of the sequentially computed logrank statistic to a rescaled Brownian motion under very general censoring and participant-arrival patterns. When interim analyses are only performed at discrete times, the decision boundaries based on continuously monitoring the logrank statistic are known to be overly conservative. This deficiency is addressed by group-sequential and α-spending methods, which, using knowledge of the interim analysis times relative to a predefined maximum number of events, allow for tighter decision boundaries [25, 22, 14]. These sequential methods allow several interim looks at the data to stop for efficacy (if the treatment shows to be beneficial) or futility (if the study is no longer likely to reach statistical significance).

Despite the profound impact that these methods have had in statistical practice, the requirement of a maximum sample size limits the utility of a promising but nonsignificant study once the maximum sample size is reached. Because of their design, extending such a trial makes it impossible to control their type-I error. Moreover, the evidence gathered in new—possibly unplanned—trials cannot be added in a typical retrospective meta-analysis, when the number of trials or timing of the meta-analysis are dependent on the trial results. Such dependencies introduce accumulation bias and invalidate the assumptions of conventional statistical procedures in meta-analysis [34]. In order to address these deficiencies, we look for flexible anytime-valid methods that provide type-I error control in two situations: (1) optional stopping, which refers to halting the experiment earlier or later than planned under arbitrary stopping rules, and (2) meta-analysis and optional continuation, which refers to the aggregation of evidence of possibly interdependent studies. Just as the existing methods, our approach is connected to early work by H. Robbins and collaborators [5, 16]. Most notably, existing approaches come with fixed stopping rules, which are not desirable in the use cases that are of our present interest. The details of the present approach are very different, and to some extent, as we will see, more straightforward.

The main result of this work is the anytime-valid (AV) logrank test, an anytime-valid test for the statistical comparison of time-to-event data from two groups of participants. The AV logrank test uses the exact ratio of the sequentially computed Cox partial likelihood as test statistic. This is in contrast to the conventional logrank test, that can be interpreted as transforming a test statistic into a p-value whose distribution is, in all applications we are aware of, not determined exactly but rather approximated by a normal or a generalized beta distribution [24]. Such approximations are valid only in asymptotic senses, and even then, some further conditions must often hold. In contrast to these approximations, the exact anytime-valid logrank test has guaranteed type-I error control also for small sample sizes, without worries whether asymptotic approximations to the sampling distribution are justified. The advantage of having an exact test becomes particularly clear in the case of unbalanced allocation, when both control and treatment groups start with different numbers of participants. For this case, it has been documented that α-spending approaches do not provide strong type-I error guarantees due to the approximations involved [51]. The basic version of the AV logrank test is, however, exact; unbalanced allocation presents no difficulties.

From a technical point of view, we show, under general patterns of incomplete observation, that under the composite null hypothesis our test statistic is a continuous-time martingale with expected value equal to one. Statistics with this sequential property are referred to as test martingales; they form the basis of anytime-valid tests [28]. The AV logrank test is a concrete instance of such a test martingale derived from the recent theory of anytime-valid hypothesis testing based on E-processes [29, 9, 37, 46]. At each time an event takes place, it takes on the form of a likelihood ratio ratio between two multinomial distributions in a sampling-without-replacement setting. Together with the underlying assumption of continuous time, this without-replacement property makes the test martingale and corresponding filtration different in spirit than in most existing work (such as Lindon and Malek [19] who consider discrete test martingales for multinomial experiments under sampling-with-replacement). Nevertheless, just as in [48, 49] who also considered a sampling without-replacement (but otherwise quite different) setting, we can show similar results on anytime-validity as in discrete-time, sampling-with-replacement settings.

In contrast to p-values, an analysis based on E-processes can extend existing trials as well as inform the decision to start new trials and meta-analyses, while still controlling type-I error rate. Type-I error control is retained even (i) if the E-process is monitored continuously and the trial is stopped early whenever the evidence is convincing, (ii) if the evidence of a promising trial is increased by extending the experiment and (iii) if a trial result spurs a new trial with the intention to combine them in a meta-analysis.

The AV logrank test was developed with a specific application in mind and it illustrates its usefulness. Some of the authors were involved in applying the AV logrank test to the continuous meta-analysis of seven Coronavirus disease (COVID-19) clinical trials—the results are available as a living systematic review including code and summary data to reproduce the analysis [33]. This analysis was performed concurrently with the trials in a so-called Anytime Live and Leading Interim (ALL-IN) meta-analysis [35]. We remark that even in the presence of dependencies between the existence and size of the trials, the test based on the multiplication of the values of E-processes retains type-I error control as long as all trials test the same global null hypothesis, as was the case in the above application. This is generally useful if we want to combine the results of several trials in a bottom-up retrospective meta-analysis, where ‘bottom-up retrospective’ means that the trials did not happen in coordination, and the decision to group them together was made after they had already started, so that no coordinated ‘top-down’ stopping rule can be enforced. It is even possible to obtain an interim meta-analysis result by combining interim results of ongoing trials by multiplication, stepping beyond the realm of existing sequential approaches.

We begin by describing the hypothesis that is being tested, the data that are available, and Cox’ proportional hazards model. We are interested in comparing the survival rates between two groups of participants, Group A and Group B. In a randomized controlled trial, Group A would signify the control group; Group B, the treatment group. We assume that the available data about m participants are a subset of { ( X i , g i , δ i ) : i = 1 , … , m }, where X i = min { T i , C i } is the minimum between the event time T i and the (possibly infinite) censoring time C i ; g i is a zero-one covariate depending on group membership ( g i = 0 signifies that i ∈ A; g i = 1, that i ∈ B); and δ i = 1 { X i = T i } is the indicator of whether the event was witnessed before censoring or not. The idea is that at (continuous) time t, we have observed all (and only) those tuples ( X i , g i , δ i ) with X i ≤ t, i.e. those participants that have experienced an event. Let m A be the number of members of Group A and m B the number of members of Group B—then m A + m B = m. Define g = ( g 1 , … , g m ), the vector of group memberships. We assume that T 1 , … , T m , C 1 , … , C m are independent and have continuous distribution functions. The continuity assumption precludes tied observations; we relax this assumption later on, in Section 3.3. For i = 1 , … , m, the survival rates are quantified by the hazard functions λ i = ( λ t i ) t ≥ 0 for T i , given by (2.1) λ t i = − d d t ln P { T i ≥ t } . As is customary, the hazard function λ i at t can be interpreted via the conditional probability of witnessing an event in a short time span provided that the event has not been witnessed up to t, that is, (2.2) P { t ≤ T i < t + Δ t | t ≤ T i } = λ t i Δ t + o ( Δ t ) as Δ t → 0 . Given our interest in comparing the survival rates between the two groups, suppose that all participants i of Group A have a common hazard function λ t i = λ t A ; members i of Group B, λ t i = λ t B (thus, for all participants i within the same group, the event times T i are identically distributed). Using the data, we wish to test proportional hazards hypotheses. Concretely, we test the hypotheses H 0 that the hazard function of the members of both groups satisfy λ t A = θ 0 λ t B , against an alternative hypothesis H 1 that λ t B = θ λ t A for a θ ≠ θ 0 . As a first application of the methods that we develop, we consider a left-sided alternative, that is, (2.3) H 0 : λ t B = θ 0 λ t A vs. H 1 : λ t B = θ λ t A for some θ ≤ θ 1 < θ 0 and all t , where θ is known as the hazard ratio and is the main quantity of statistical interest, and θ 1 would be, in a clinical trial, a minimal clinically relevant effect size. The alternative is what we hope for in case of negative events, such as death, with treatments that are set out to lower (relative to the control condition) the hazard rate. Notice that the hypotheses in (2.3) are, in fact, nonparametric. Similarly, if the event is positive, e.g., recovery from an infection, we would typically set a right-sided alternative, which can be also be treated with the present methods. Two-sided alternatives and also the full alternative hypothesis H 1 ′ : θ ≠ θ 0 are also amenable to the methods that will follow. We remark, however, that all the methods retain their type-I error guarantees irrespective of the specific alternative that we use. As will be seen in Example 3.2 below, this is of particular interest in the simple, yet important case with θ 0 = 1 and full alternative H 1 ′ : θ ≠ 1—in that case, we can think of our test as really testing ‘the treatment has no effect’ against general alternative ‘the treatment has an effect’ (even though the proportional hazard assumption may be substantially violated).

We now turn to defining Cox’ partial likelihood PL t , which is at the center of our approach. To that end, we need a battery of standard definitions—we lay them out to establish the notation. Let y t i = 1 { X i ≥ t } be the at-risk process, that is, the indicator of whether participant i is still at risk at time t, and let y ¯ t A = ∑ i ∈ A y t i and y ¯ t B = ∑ i ∈ B y t i be the number of participants at risk in each of the groups at time t. Define y t = ( y t 1 , … , y t m ), the vector of at-risk processes, and R t = { j : y t j = 1 }, the set of participants at risk at time t. Let T ( 1 ) < T ( 2 ) < ⋯ < T ( N ¯ ∞ ) be the set of ordered events times that were witnessed (not censored). Note that, if all participants witness the event and censoring is absent, N ¯ ∞ = m. For each k = 1 , … , N ¯ ∞ , let I ( k ) be the index of the individual that witnessed the event at time T ( k ) . This means, for example, that if participant with label three was the fifth to witness the event, then I ( 5 ) = 3. Abbreviate by y ( k ) i , y ¯ ( k ) A , y ¯ ( k ) B , R ( k ) the corresponding quantities at event time T ( k ) , and define g ( k ) : = g I ( k ) . Cox’ partial likelihood PL θ , t can be sequentially computed by (2.4) PL θ , t = ∏ k : T ( k ) ≤ t θ g ( k ) ∑ l ∈ R T ( k ) θ g l = ∏ k : T ( k ) ≤ t θ g ( k ) y ¯ ( k ) A + θ y ¯ ( k ) B . Cox’ likelihood evaluated at the event times T ( 1 ) , T ( 2 ) , … coincides to that of a sequence of multinomial trials where, at event time T ( k ) , each of the participants i ∈ R ( k ) witnesses the event with probability (2.5) p θ , ( k ) ( i ) : = P { I ( k ) = i ∣ y ( l ) , g ; l = 1 , … k } , p θ , ( k ) ( i ) = θ g i y ¯ ( k ) A + θ y ¯ ( k ) B . We note that if, under the null, the groups are indistinguishable (see also Example 3.2 below), θ = θ 0 = 1, (2.4) reduces to the likelihood of a sequence of draws-with-replacement from an urn that initially contains m A marbles of color A and m B marbles of color B: (2.6) PL θ , t = ∏ k : T ( k ) ≤ t θ g ( k ) y ¯ ( k ) A + y ¯ ( k ) B . Cox showed that, indeed, conditionally on all the information accrued strictly before T ( k ) , the probability that participant i observes an event at time T ( k ) is exactly p θ , ( k ) ( i ) as long as the hazard ratio is θ. With these likelihood computations at hand, we are in place to show the main contribution of this article, the AV logrank test, which uses the partial likelihood ratio as the test statistic.

In this section the AV logrank test for (2.3) is introduced; its type-I error guarantees and optimality properties are investigated. We give a solution to the first of the purposes laid down in the introduction: we show that the AV logrank test is anytime valid—its type-I error guarantees are not affected by optional stopping. The fact that it is also type-I-error-safe under optional continuation, our second purpose, is proven in Section 5. Without further ado, we define the AV logrank statistic S θ 0 , t θ 1 , typically, θ 0 = 1, for (2.3) as the partial likelihood ratio (3.1) S θ 0 , t θ 1 = PL θ 1 , t PL θ 0 , t = ∏ k : T ( k ) ≤ t p θ 1 , ( k ) ( I ( k ) ) p θ 0 , ( k ) ( I ( k ) ) . Here, p θ , ( k ) is as defined in (2.5); the product that defines our statistic S θ 0 , t θ 1 runs over the events that have been witnessed up to and including time t, and the empty product is taken to be equal to one. As is conventional with likelihood ratios, high values of S θ 0 , t θ 1 are indicative that the alternative hypothesis is better than the null hypothesis at the describing the data. Given a tolerable type-I error bound α and an arbitrary random time τ, the AV logrank test is the test that rejects the null hypothesis if S θ 0 τ θ 1 is above the threshold 1 / α, that is, (3.2) ξ θ 0 , τ θ 1 = 1 { S θ 0 , τ θ 1 ≥ 1 / α } : = 1 if S θ 0 , τ θ 1 ≥ 1 / α 0 otherwise . As we will see, by its sequential properties, S θ 0 , t θ 1 takes large values with small probability under the null hypothesis uniformly over time, which translates into type-I error control for the test ξ θ 0 , τ θ 1 . This observation is behind the any-time validity of the AV logrank test, and of anytime-valid tests in general (more details and general constructions to the effect of anytime-valid sequential testing can be found in the work of Ramdas et al. [28]). We show in the following proposition that the test ξ θ 0 , τ θ 1 has the desired type-I error control.

This result can be readily obtained using the sequential-multinomial interpretation of Cox’ likelihood ratio. As we will see, in Section 3.1, this result can be interpreted in terms of E-variables and E-processes [9]. Define the process ( S θ 0 , ( k ) θ 1 ) k = 1 , 2 , … as the value of the AV logrank statistic at the event times T ( k ) , that is, S θ 0 , ( k ) θ 1 : = S θ 0 , T ( k ) θ 1 . In this time discretization, the AV logrank statistic is the product of random variables (3.3) R θ 0 , ( k ) θ 1 = p θ 1 , ( k ) ( I ( k ) ) / p θ 0 , ( k ) ( I ( k ) ) , the one-outcome partial likelihood ratio for the kth event, where p θ 0 , ( k ) is as in (2.5) and k = 1 , 2 , … .

Proposition 3.1 is a variation of the standard statement and result establishing that tests based on E-processes are anytime-valid [29]. As remarked by a referee, one notable difference is that, within a clinical trial setting, in the standard statement each subsequent factor in S θ 0 , t θ 1 would refer to an additional participant coming in, whereas in our setting, it corresponds to an event, meaning that a patient leaves. Nevertheless, the technique used to prove Proposition 3.1 is entirely analogous to the proof of the standard result (namely, Ville’s inequality), as has previously shown to be also the case in other anytime-valid work that deals, like ours, with a drawing-without-replacement setting [48, 49].

Under general patterns of incomplete observation—like independent censoring or independent left truncation—, the AV logrank test provides the same type-I error guarantees. To prove this we do need to go beyond the standard, discrete-time setting: we give an alternative proof of Proposition 3.1 in Appendix A using the counting-process formalism [1]. There, we show that if the compensators of the underlying counting processes have a certain general product structure—which is the case under complete observation—, the AV logrank test is anytime-valid. We then refer to Andersen et al. [1], who show that this structure is preserved under said patterns of incomplete observation.

The AV-logrank test is optimal—in a sense to be defined in the next section—among a large family of statistics. A second look at the proof of Proposition 3.1 suggests a generalization of the AV logrank statistic given in (3.1). Let, for each k, q ( k ) be a probability distribution on participants in the risk set R ( k ) which is only allowed to depend on y ( 1 ) , … , y ( k ) . Analogously to (3.3), we define the one-outcome ratio R θ 0 , ( k ) q : = q ( k ) ( I ( k ) ) / p θ 0 , ( k ) ( I ( k ) )—we now use q ( k ) instead of p θ 1 —, and (3.5) S θ 0 , t q : = ∏ k : T ( k ) ≤ t R θ 0 , ( k ) q = ∏ k : T ( k ) ≤ t q ( k ) ( I ( k ) ) p θ 0 , ( k ) ( I ( k ) ) . A modification of the previous argument shows, for any random time τ, a type-I error guarantee for the test ξ θ 0 , τ q based on the value of S θ 0 , τ q , that is, ξ θ 0 , τ q : = 1 { S θ 0 , τ q ≥ 1 / α } (see Proposition 3.1). Any such test is also anytime valid as long as each q ( k ) depends on the data only through y ( 1 ) , … , y ( k ) . In Section 3.2, we use this generalization to provide tests when no value of θ 1 is available. This generalization raises a natural question about the optimality of the AV logrank test based on (3.1) among test statistics of the form (3.5). This is the subject of the next section.

In this section we present an approximation to the AV logrank test introduced in the previous section. This is based on a Gaussian approximation to the logrank statistic. The approximation is of interest for two reasons. First, in practical situations, only the logrank Z-statistic (a standardized form of the classic logrank statistic) and other summary statistics may be available—and not the full risk-set process. This is often the case in medical meta-analyses, where the full data sets (Individual Participant Data, IPD, for each trial) are often not available, but summary statistics are. The Gaussian approximation to the AV logrank statistic can then often still be used (we remark though that in the BCG trial meta-analysis [33] referred to in the Introduction, there was IPD access and the exact anytime-valid logrank test was performed). The second reason, which we address in Section 4.1, is related to the fact that α-spending and group-sequential approaches, which we use as benchmarks, are also based on Gaussian approximations to the classic logrank statistic. Consequently, the behavior of the Gaussian approximation gives further insights into how the AV logrank statistic compares to group-sequential and α-spending approaches as well. We henceforth focus on θ 0 = 1, a central case of interest also considered in Example 3.2.

Our general strategy is close in spirit to that followed in the construction of the exact AV logrank statistic in Section 3. We build likelihood ratios using a classic Gaussian approximation for the distribution of the original logrank statistic [32]. If the distribution of this statistic was exactly normal, we could monitor continuously its likelihood ratio. We show through extensive simulation in which regimes this approximation behaves similarly to the AV logrank statistic.

We begin by recalling the definition of the Z-score associated to the classic logrank test. Let E i B = O i p i B with p i B = y ¯ i B / ( y ¯ i A + y ¯ i B ) be the expected (under the null) number of events witnessed in the time interval ( t i − 1 , t i ] in group B, and let V i B = O i p i B ( 1 − p i B ) y ¯ i − O i y ¯ i − 1 be its variance. After k observations the Z-score associated to the classic logrank statistic, Z k , is given by (4.1) Z k = ∑ i ≤ k O i B − E i B ∑ i ≤ k V i B . The numerator in the definition of Z k is the classic logrank statistic H k = ∑ i ≤ k O i B − E i B , which is typically interpreted as the cumulative difference between observed counts O i B and the expected counts E i B in Group B. The factor y ¯ i − O i y ¯ i − 1 found in V i B can be interpreted as a multiplicity correction, that is, a correction for ties [15, p. 207]. When only one event is witnessed between two consecutive observation times, then O i = 1, E i B = p i B , and V i B = p i B ( 1 − p i B ). We remark that the above formulation is also found in the work of Cox [3, (26)].

Fix some initial m A , m B and let γ = m B / m A . Schoenfeld [32] showed that in the limit for m A → ∞, m B = γ m A , k → ∞, k / m A → 0, and log θ = O ( ( m A + m B ) − 1 / 2 ), the distribution of Z k under any distribution with hazard ratio θ 1 converges to the distribution of a normal with unit variance and mean (4.2) μ 1 = log ( θ 1 ) m B m A / ( m A + m B ) 2 . Appendix C provides more details about this result. If Z k were exactly N ( μ 1 , 1 ) distributed, then an easy calculation shows that the logrank partial likelihood ratio statistic S 1 , k θ 1 would be equal to a likelihood ratio of two Gaussians, given by (4.3) S k G : = exp − 1 2 N ¯ k μ 1 2 + N ¯ k μ 1 Z k , where N ¯ k is the total number of observations up until time t k and μ 1 is given by (4.2). We thus put forward S k G as a Gaussian approximation to the logrank statistic S 1 , k θ 1 . For an arbitrary random observation time t K ∈ { t 1 , t 2 , … }, we refer to the test ξ K G = 1 { S K G ≥ 1 / α } as the Gaussian AV logrank test for (2.3). Recall that we test θ 0 = 1, which corresponds to the asymptotic mean of the Z-score under the null hypothesis being μ 0 = 0. Clearly, the approximation may fail in practice for at least two reasons: first, m A , m B , k and θ 1 may not fall under Schoenfeld’s asymptotic regime; second, Schoenfeld’s result has been derived in a nonsequential setting, whereas we aim to use it in a sequential setting with optional stopping. Thus, in Appendix C.1 extensive simulations are performed to show in which regimes the Gaussian logrank test retains type-I error guarantees. In Appendix C.2, it is shown that, under continuous monitoring, the Gaussian AV logrank test tends to be more conservative—it needs more data than the exact one. The conclusion is the following: S K G can be used for designs with balanced allocation, and it approximates S 1 , K θ 1 well for hazard ratios between 0.5 and 2.

We now compare the rejection regions defined by the Gaussian logrank test to those of continuously monitoring using α-spending and group-sequential approaches.