In phase II trials with binary efficacy endpoints, let p E and p C denote the true response rates for the experimental arm and the control arm, and p ˆ E and p ˆ C denote the observed response rates. We aim to address the two-sample test with the following hypotheses: H 0 : p E = p C = p 0 , H a : p E = p 1 , p C = p 0 , p 1 > p 0 . Traditional approaches for the two-sample test rely solely on between-arm comparisons and yield only binary trial outcomes: either rejecting or accepting H 0 . To reduce the required sample size and draw more meaningful conclusions from two-sample tests, we combine one-sample rejection decision rules with traditional two-sample rules and introduce a statistically inconclusive region.

In a one-stage setting, assume n E patients are recruited to the experimental arm and n C patients are recruited to the control arm, leading to a total of N = n E + n C patients. Let y E and y C denote the number of patients demonstrating successful responses in the experimental and control arm, respectively. The decision rules for reaching one of the three outcomes of rejecting H 0 , rejecting H a , or rejecting neither are defined as follows: (2.1) If p ˆ E − p ˆ C ≥ p s ∩ p ˆ E ≥ p m , reject H 0 ; If p ˆ E − p ˆ C < p s , reject H a ; If p ˆ E − p ˆ C ≥ p s ∩ p ˆ E < p m , declare statistically inconclusive , where p s denotes the statistical significance boundary ( p s > 0), and p m denotes the clinical relevance boundary ( p m > 0). Given a 1:1 randomization, the decision rules in Equation (2.1) can be simplified by assuming n E = n C = N / 2, that is If y E − y C ≥ s ∩ y E ≥ m , reject H 0 ; If y E − y C < s , reject H a ; If y E − y C ≥ s ∩ y E < m , declare statistically inconclusive , where s = N 2 p s , and m = N 2 p m . Here we refer to this design as a 2-by-2 TDR design, where the decision rules for both statistical significance and clinical relevance each contain two regions: y E − y C ≥ s or y E − y C < s for statistical significance, and y E ≥ m or y E < m for clinical relevance. Under the independent and normality assumption, the conditions of the decision rules are equivalent to constructing two z-test statistics.

The statistically inconclusive region is reserved for the situation where there are substantial differences between the experimental arm and the control, while the observed responses in the experimental arm are suboptimal in terms of the historical control rate. This may occur when the trial population differs from the population used to derive the historical control rate. In this case, the clinical decision regarding whether to proceed to a phase III trial or terminate the current trial requires more deliberation. Primary investigators and statisticians should comprehensively review factors such as regulatory requirements, commercial potential, and practicality of administering the treatment, in order to decide on warranting further investigations.

In our one-stage TDR design, we use exact binomial probabilities to calculate the type I error α, type II error β, and inconclusive region probabilities η under H 0 and γ under H a , as follows: α = ∑ D 1 ∩ D 2 B ( y E , n E , p E ∣ H 0 ) B ( y C , n C , p C ∣ H 0 ) , β = ∑ D 1 ′ B ( y E , n E , p E ∣ H a ) B ( y C , n C , p C ∣ H a ) , η = ∑ D 1 ∩ D 2 ′ B ( y E , n E , p E ∣ H 0 ) B ( y C , n C , p C ∣ H 0 ) , γ = ∑ D 1 ∩ D 2 ′ B ( y E , n E , p E ∣ H a ) B ( y C , n C , p C ∣ H a ) , where D 1 = { ( y E , y C ) : y E − y C ≥ s }, D 2 = { y E : y E ≥ m }, and D 1 ′ and D 2 ′ denote the complementary sets of D 1 and  D 2 . In addition, we introduce λ = ( η + γ ) / 2 as the expected inconclusive probability. This definition is based on a common assumption that the unknown true response rates of both arms vary uniformly between the null and alternative hypotheses. The statistically inconclusive regions under H 0 and H a can be controlled simultaneously by constraining γ and λ instead of γ and η to prevent highly imbalanced inconclusive regions under H 0 and H a . To provide finer control over inconclusive regions, it is possible to introduce both upper and lower boundaries for determining the statistical significance. This extension is referred to as a 3-by-2 TDR design, with three regions for statistical significance (i.e. y E − y C ≥ s, r < y E − y C < s, or y E − y C ≤ r) and two regions for clinical relevance (i.e. y E ≥ m or y E < m) in the decision rules. A detailed description is provided in Section 1.1 of the Supplementary Materials.

In this section, we consider extending the proposed design to a two-stage trial setting, for the purpose of ethically stopping a trial early given insufficient evidence of efficacy. In stage 1, we enroll and randomize n C 1 patients to the control arm and n E 1 patients to the experimental arm. If the trial is not stopped early, we proceed to stage 2, where n C 2 and n E 2 patients are randomized to each arm. We denote N 1 = n C 1 + n E 1 and N 2 = n C 2 + n E 2 as the total sample size in stages 1 and 2, respectively. The number of responses observed in the stage 1 (or stage 2) are denoted as y E 1 and y C 1 (or y E 2 and y C 2 ) for the treatment and control arms, respectively. At the conclusion of stage 1, an interim analysis is performed, and the trial proceeds to stage 2 if (2.2) y E 1 − y C 1 > s 1 and y E 1 ≥ m 1 , where s 1 is a statistical difference threshold for early stopping and m 1 is a clinical relevance threshold for early stopping. Note that the inconclusive regions are excluded in the interim analysis for simplicity. We denote the probabilities of proceeding to stage 2 under H 0 and H a as Pr ( S 1 ∣ H 0 ) and Pr ( S 1 ∣ H a ), respectively, where S 1 = { n E 1 , n C 1 , s 1 , m 1 : y E 1 − y C 1 > s 1 ∩ y E 1 ≥ m 1 } represents the condition (2.2). In our design, we propose to control Pr ( S 1 ∣ H 0 ) ≤ 1 − e s 0 and Pr ( S 1 ∣ H a ) ≥ 1 − e s 1 , where e s 0 and e s 1 are early stopping probability levels under the null and alternative hypotheses, respectively. In this paper, we set e s 0 = 0.50 and e s 1 = 0.05 as reasonable constraints.

In stage 2, the proposed design will proceed as described in Section 2.1, with type I error α, type II error β, and inconclusive region probabilities η under H 0 and γ under H a defined as below: α = ∑ D 1 ∩ D 2 B ( y E 2 , n E 2 , p E ∣ H 0 ) B ( y C 2 , n C 2 , p C ∣ H 0 ) Pr ( S 1 ∣ H 0 ) , β = ∑ D 1 ′ B ( y E 2 , n E 2 , p E ∣ H a ) B ( y C 2 , n C 2 , p C ∣ H a ) Pr ( S 1 ∣ H a ) , η = ∑ D 1 ∩ D 2 ′ B ( y E 2 , n E 2 , p E ∣ H 0 ) B ( y C 2 , n C 2 , p C ∣ H 0 ) Pr ( S 1 ∣ H 0 ) , γ = ∑ D 1 ∩ D 2 ′ B ( y E 2 , n E 2 , p E ∣ H a ) B ( y C 2 , n C 2 , p C ∣ H a ) Pr ( S 1 ∣ H a ) , where D 1 = { ( y E 2 , y C 2 ) : y E 2 − y C 2 ≥ s 2 − ( y E 1 − y C 1 ) }, D 2 = { y E 2 : y E 2 ≥ m 2 − y E 1 }, s 2 is the statistical significance boundary ( s 2 > y E 1 − y C 1 ), and m 2 is the clinical relevance boundary ( m 2 > y E 1 ).

With the introduction of the inconclusiveness region, the power of the TDR design is defined as π = 1 − β − γ, which is the probability of rejecting H 0 when H a is true. Given a specific minimum target power π min and maximum type II error level β max , we control the inconclusive region probabilities γ and λ under their maximum allowable constraints γ max and λ max . As a result, there might be multiple sets of ( α , β , γ , λ ) satisfying these requirements. For example, given π min and β max , the maximum allowable inconclusive probability under H a is given by γ max = 1 − ( β max + π min ). To ensure proper control of the trial’s inconclusive probabilities, a possible γ should not exceed this threshold (i.e., γ ≤ γ max ). Similarly, λ max is given by λ max = ( η max + γ max ) / 2, where η max represents the maximum target level for the inconclusive probability η. To facilitate parameter search under these constraints, we propose a loss function for systematic evaluation, which is discussed in Section 2.4.

The sample size for the proposed one-stage TDR design is determined using a two-step approach. Firstly, for each candidate sample size, sets of parameters { s , m , n C , n E } are obtained through an incremental search over a grid of design parameters { ( α , β , γ , λ , π ) : α ≤ α max , β ≤ β max , γ ≤ γ max , λ ≤ λ max , π ≥ π min }, where α max denotes the maximum target type I error level. For each given set of ( α , β , γ , λ , π ) satisfying these constraints, we search for all possible pairs of s and m within pre-defined searching regions. Reasonable regions for s and m can be { s : s ∈ [ p C ( n E − k n C ) , p E n E − k p C n C + 4 σ pool ] , − n C ≤ s ≤ n E } , { m : m ∈ [ p C n E , p E n E + 4 σ E ] , m ≤ n E } , where k is the randomization ratio, σ pool is the pooled standard error, and σ E is the standard error under H a . The optimal ( s , m ) is then obtained by selecting the pair with the smallest type I error α. Secondly, among all candidate sample sizes, the optimal design parameters ( α , β , γ , λ , π ) and the corresponding sample size are selected through a loss function that balances the trade-off between trial power and sample size. We provide a systematic evaluation of sample size and power using the loss function described in Section 2.4. The parameters yielding the smallest loss score are then selected as the optimal parameters.

The proposed one-stage design is expected to reduce type I error by introducing the inconclusive region, compared to the design by [10], at the same sample size. This occurs when the difference in the number of responses between the two arms is larger than s, but the number of observed responses in the experimental arm is less than m. Theoretically, given a fixed sample size N, the relationships between ( s , m ) and the associated error rates (or inconclusive probabilities) can be summarized as follows: when m is kept constant, increasing s reduces α, increases β, and reduces γ and η; when s is kept constant, increasing m reduces α, increases γ and η, and has no effect on β. Under H 0 , the criterion of clinical relevance has less effect compared to its effect under H a . This is because, when p E = p C , it is more likely that y E < m. For the same reason, η is generally larger than γ.

Previous studies have established the practice of optimizing trial design using loss functions, such as [8, 12, 16], among others. In this design, we propose using a loss function to systematically evaluate the effects of inconclusive region constraints to optimize both sample size and power. For each sample size and its corresponding optimal design parameters ( α , β , γ , λ , π ), we calculate a loss score at the sample size n and power π with respect to a reference sample size n 0 and a reference power π 0 by a loss function L ( n , π , n 0 , π 0 ). The reference sample size n 0 is the sample size per arm calculated using a standard two-group sample size calculation under the same hypothesis as the TDR design. The reference power π 0 is the corresponding power calculated in the reference sample size method, i.e. the probability of correctly accepting the alternative hypothesis of the reference sample size method. The primary principle of the loss function is to penalize an increase in sample size and a reduction in power. According to [11], a loss function should meet the following criteria: 1.

Monotonicity: at a fixed sample size n or a fixed power π, the loss score should monotonously increase if power decreases or sample size increases. L ( n , π 1 , n 0 , π 0 ) > L ( n , π 2 , n 0 , π 0 ) ⇔ π 1 < π 2 ; L ( n 1 , π , n 0 , π 0 ) > L ( n 2 , π , n 0 , π 0 ) ⇔ n 1 > n 2 .

Based on the criteria discussed above, we propose the following loss function: L ( n , π , n 0 , π 0 ) = σ ( w n n 0 + ( 1 − w ) π 0 π − 1 ) , where w is a weighing parameter, and σ ( · ) is a link function defined as σ ( x ) = 1 1 + e − x . The link function σ ( · ) scales the loss score to the range of ( 0 , 1 ). The parameter w determines the trade-off between reducing sample size and increasing power. We performed a sensitivity analysis on w and found that the sample size and power were invariant to w when w > 0.4 (Supplementary Figure S1). Within the range of w ≤ 0.4, a larger w gives greater priority to reducing the sample size, while a smaller w prioritizes increasing power. Therefore, we recommend using w = 0.5, which assigns equal importance to sample size reduction and power improvement and provides an optimized balance between sample size and power. We recommend calculating n 0 using the standard sample size formula for testing H 0 : p E − p C = 0 and H a : p E − p C > 0 [2]. When the sample size is smaller than that of the standard two-sample test and the power is greater, which is the most desirable scenario, the sum of the first two components is smaller than 1, resulting in a loss score smaller than 0.5; when the sample size and power match those of the standard two-sample test, the loss score equals 0.5; when the sample size is larger and the power is lower, the loss score is greater than 0.5, which is considered undesirable.

The optimal parameters for the inconclusive regions are selected as the smallest solution set that minimizes the loss function. Firstly, given minimum target power π min and sample size n, we specify a pair of ( γ max , λ max ). Then the optimal design sample size and the corresponding power ( N ∗ , π ∗ ) are determined by minimizing the loss score. Formally, (2.3) ( N ∗ , π ∗ ) = arg min N ∈ Q L ( N , π , n 0 , π 0 | γ max , λ max ) , where Q represents the search set for the sample size. An example illustrating the determination of γ max and λ max is provided in Table S1 in the Supplementary Materials. In practice, we impose an additional constraint on power π to ensure that it remains at a relatively high level, requiring π ≥ π min − c, where c is a constant (e.g., c = 0.05). When comparing different trial configurations, the loss score provides a systematic evaluation of both sample size and power.

Table 1 lists the optimal TDR one-stage 2-by-2 design parameters with varying differences in response rate, δ = p E − p C ∈ { 0.15 , 0.20 , 0.25 }, under target levels α max = 0.20, β max = 0.20, π min = 0.80, and c = 0.05. In this table, γ max and λ max are design parameters corresponding to the optimal sample size, determined using the loss function with a weight parameter w = 0.50, which equally weighs the importance of sample size and power. For each case of response rates, we employ a 1:1 randomization. The total sample size required for the trial is denoted by N with corresponding decision boundaries s and m. To evaluate the proposed design, we compare the operating characteristics of the TDR design with the method proposed by [6] (HW) and the method proposed by [10] (LBR). The comparison evaluates the percentage reduction in sample size relative to the conventional calculation for two-sample proportions under the same settings of type I error α max , and type II error β max . The results are shown in Figure 1. Overall, the TDR design provides 26.7–51.7% sample size savings as compared to the conventional approach, while the HW method provides up to 22.7% sample size savings and the LBR method provides 20.0–42.6% sample size savings. In most of the scenarios, the proposed method outperforms the HW method and the LBR method with minimal loss of power. The TDR method generally yields higher power than the HW method and provides more sample size savings in all cases. Compared to the LBR method, the TDR method provides superior or comparable sample size reduction up to 13.8% except in one case. TDR achieved 0.3–12.3% type II error reduction at the cost of up to 6.3% power loss and gained 1.0% power in one case. In terms of the type I error, all three methods are comparable and are constrained below 0.20. In the one case where the TDR method does not show sample size saving compared to the LBR method, the type II error is 7.5% lower and the power is 1.1% higher at response rates of (0.30, 0.50). In four of the twenty cases, the HW method does not exhibit substantial sample size saving and is therefore not displayed.

Under more stringent requirements on type I and type II errors, i.e., α max = 0.10, β max 0.10, π min = 0.80, and c = 0.05, superior performances of the proposed method are observed in most cases. Table 2 provides details of the optimal trial design parameters. Comparisons of sample size reduction, as well as operating characteristics, are shown in Figure 2. Overall, the TDR design achieves 37.2–57.0% sample size reduction, the HW design achieves up to 34.0% sample size reduction, and the LBR design achieves 37.2–49.1% sample size reduction. The TDR method provides additional 2.9–18.0% sample size savings as compared to the LBR method in 18 of the 20 cases. In two cases where equal sample sizes are calculated, TDR provides 3.1% and 1.2% reduction in type II errors at response rates of (0.20, 0.45) and (0.65, 0.85), respectively.

Extending the method to a two-stage design, we provide details of the TDR two-stage 2-by-2 design in Table 3. In addition, given the established sample size saving performance of the LBR method proposed by [10], we use it as a reference to benchmark the proposed method in terms of expected sample size (EN) and maximum sample size as shown in Figure 3. The total sample sizes required for stage 1 and stage 2 of the trial are denoted by N 1 and N 2 , respectively, with corresponding decision boundaries s 1 and m 1 for stage 1, and s 2 and m 2 for stage 2.

In a two-stage TDR design, the inconclusive region is considered only in stage 2. As a result, if the sample size in stage 1 is much larger than in stage 2, the potential for sample size savings will be limited. In 11 of the 20 cases, our proposed method provides a 2.8–8.7% reduction in expected sample size and a 5.4–15.8% reduction in maximum sample size as compared to the LBR method. In all cases, the proposed method shows reductions in type II errors compared to the LBR method. In the nine cases of no sample size reduction, we observe reductions in type II errors except for minimal type II error inflation in two cases (0.02 and 0.003 at the response rates (0.20, 0.40) and (0.35 and 0.60), respectively). This could be due to over-constraining design parameters, which could be remedied by a more granular search of γ max and λ max in regions around the current constraints, adjusting the loss function weight parameter w, or inspecting the sculpting boundaries. For example, at p C = 0.35, p E = 0.60, by decreasing the statistical difference boundary, s 2 , one could reduce the total sample size by two in the second stage with a 1.8% reduction in type I error with a power still higher than 0.75.

As previously shown, the TDR design can be applied to more stringent requirements on type I and type II errors. To account for the effect of variation in control response rates, we apply the type I error constraints to the maximum of type I errors yielded from a confidence interval of p C . A confidence interval of 30% is chosen for demonstration purposes, as with historical data, there could be a fairly informed estimation of control response rates. In general, the proposed design shows superior performance to the HW design, the LBR design, as well as the conventional design. The details can be found in Table S2 and Figure S2 of the Supplementary Materials.

We also conducted a comparison between the proposed 2-by-2 design and the 3-by-2 design. Further details can be found in Table S3 and Figure S3 in the Supplementary Materials. Overall, with more granular control of the inconclusive region, the 3-by-2 TDR design provides a reduction in type II error, which is an advantage of using the 3-by-2 design. Depending on true response rates, the 3-by-2 design may provide sample size savings in some cases. However, one should also take note of the increased design complexity when choosing between 2-by-2 and 3-by-2 designs.