The New England Journal of Statistics in Data Science

The recent advancement in genomics sequencing and molecular biology has enabled a detailed classification of patients based on genomics alterations or molecular profiles, thus inspiring the establishment of targeted therapies [1, 5] and precision medicine [2, 3, 14]. U.S. Food and Drug Administration (FDA) provides the master protocol [29, 11] guidance that facilitates the methodological innovation that coordinates efforts to investigate treatments in more than one patient population or disease types within one master protocol [42, 30, 13]. Among the master protocols, basket trials [28, 25, 13] have become a popular design since it allows the evaluation of an investigational treatment in multiple disease cohorts in parallel and hence expedite the efficiency of clinical research. U.S. Food and Drug Administration (FDA)[38] defines a basket trial protocol as follows:

Basket trials are typically conducted in a phase II trial to provide preliminary proof-of-concept evidence for clinical validation. Some of the other unique purposes of basket studies and their application examples are discussed by Cunanan et al. [8] and Tao et al. [35]. Basket trials are predominantly designed for oncology studies. This design is utilized in [34], [16], [15], [22], [27] and [41], to name a few.

In a basket trial, patients from different baskets may be expected to have similar responses because they may share common features, such as disease stages or molecular alterations. This provides the basis for information borrowing across baskets, which is one of the key statistical advantages of basket trials. Note that different information borrowing approaches can be applied. As a well-known example, Vemurafenib [12], initially approved for melanoma with V600E BRAF mutations, was approved by the US Food and Drug Administration (FDA) for treating BRAF V600 mutation-positive Erdheim-Chester disease (ECD). The BRAF basket trial used the Simon Two-Stage design [32] that treated the responses of each basket independently as they were from separate studies and hence no information was borrowed across baskets. In another example, Vitrakvi was granted accelerated approval for treating locally advanced or metastatic solid tumors harboring a neurotrophic tyrosine kinase receptor (NTRK) gene fusion. The Vitrakvi basket trial used full information borrowing by pooling data together across all baskets, assuming that the response to the drug was homogeneous across baskets. These are the two extreme strategies for information borrowing. With the emergence of basket trials, FDA issued the general guidance on grouping strategies, stating FDA’s position on the generalizability of the results of multiple basket studies [37] and providing guidance on the overall design of master protocols, as exemplified by the BRAF V600 basket trial [38]. An excellent review and discussion on the guidance of basket trials can be found from [31]. A wide range of statistical research has been devoted to exploring more flexible methods of information borrowing. These methods dynamically or partially borrow information across baskets based on the trial data.

One class of methods attempts to borrow across all baskets. A Bayesian sequential monitoring method is presented by Simon et al. [33]. It involves multiple interim looks, with the amount of data borrowing at each interim determined by the posterior probability of homogeneity. Thall et al. [36] and Berry et al. [4] proposed the basket trial design using the full Bayesian Hierarchical Model (BHM), with the treatment effects of the baskets modeled by a distribution $\mathcal{F}$. Due to a small number of baskets in actual basket trials, the variance in $\mathcal{F}$ is difficult to be reliably estimated, leading to potentially substantial Type I error inflation [9]. To circumvent this issue, Chu and Yuan [6] suggested a Calibrated Bayesian Hierarchical model (CBHM) that provides a more reliable estimate of the variance in $\mathcal{F}$ by pre-determine the shrinkage parameter. Their results suggest that Type I errors are better controlled than BHM, especially for the cases where the effect of the basket is heterogeneous.

There is another class of methods, in which similar baskets are first clustered and the information among similar baskets is fully borrowed. Unlike the BHM model which assumes the treatment effect for each basket is exchangeable and localizes around a common mean, the exchangeability-nonexchangeability (EXNEX) model of Neuenschwander et al. [21] assumes that some baskets may be predefined to be exchangeable (EX), while some may not be exchangeable (NEX). Thus, each basket is assigned to either an EX component, which allows within component parameter to be partially exchangeable; or an NEX component which is nonexchangeable from others. The assignment is based on pre-specified probabilities. Chu and Yuan [7] and Zhou and Ji [45] proposed a more sophisticated method, in which a latent class variable is employed to group baskets into clusters and hence avoid pre-specifying the aforementioned probabilities in the EXNEX model. According to their approach, the treatment effects within each cluster are assumed to be centralized such that information is borrowed locally using the BHM; a Dirichlet distribution prior is applied to the weighting probability that assigns the baskets into the clusters; and the number of latent clusters and the basket memberships are inferred by the data through a Dirichlet Process Mixture Model (DPMM) model [19, 20]. The Mixture models have a common difficulty in choosing the number of clusters. Within the Bayesian framework, one could consider the number of clusters as an unknown parameter and specify it with a prior distribution. This kind of model is referred to as the Mixture of Finite Mixtures (MFM). Miller and Harrison [18] proved many characteristics of the DPMM are also exhibited by MFM. By applying the Markov chain Monte Carlo (MCMC) sampling algorithm similar to DPMM but with feasible alteration, this sampling algorithm exhibits higher efficiency than the reversible jump technique. Geng and Hu [10] applied MFM onto basket trials for binary endpoint.

Basket trials can have a variety of designs. First of all, it may or may not have the concurrent control arms [26]. It can also be applied for various types of endpoints: a continuous outcome design is presented by Zheng and Wason [44] and a time to event design is presented by Xu et al. [43]. Some settings also include the multiple covariates [44, 24]. In prospective circumstances, the treatment effect of each basket may depend on several therapeutic indicating covariates, and the coefficients of one covariate can vary across baskets. Interim Analyses is also popular when designing clinical trials. In this article, we primarily focus on the evaluation of a few Bayesian approaches, including Bayesian Hierarchical Model (BHM), Calibrated Bayesian Hierarchical Model (CBHM), and Mixture of Finite Mixtures (MFM), with extended the assumptions: (i) continuous endpoints; (ii) with a concurrent controlled arm; (iii) interim analysis designs and (iv) adjusting covariates effects across different baskets.

The trials to which the Bayesian approaches are applied are mostly without a concurrent placebo control. In oncology trials, non-controlled trials are usually adopted due to ethical reasons. But in many other situations, for example, when (i) side-effects and tumor size shrinkage are limited so blindness can be maintained, and (ii) patients receive the investigational treatment and placebo in addition to a current curative treatment so there are no ethical concern, randomized trials are advantageous to provide more definitive answer to treatment effects and safety to the investigational product. To emphasize the importance and the principles of estimating treatment effects and their sensitivity analysis, the US Food and Drug Administration (FDA) released “E9(R1) Statistical Principles for Clinical Trials: Addendum: Estimands and Sensitivity Analysis in Clinical Trials”[39]. This implies that assessing the magnitude of the treatment effect potentially will be of greater concern in many trial designs. As such, this article explores the performance of Bayesian methods in basket trials with a control arm and continuous endpoint. However, these methods can be readily extended to different designs.

The remaining part of this article is organized as follows. The trial setting and assumptions are described in Section 2. In Section 3, the considered Bayesian methods are introduced and discussed in detail. To assess and compare the performance of these methods, simulation studies and results are presented in Section 4. An application example is provided in Section 5. Conclusion and possible extension of the current work are addressed in Section 6. The additional results, sensitivity analysis and technical details including derivations are given in the Supplementary Material.

Consider a controlled basket trial with continuous endpoints and with K baskets. If interim analysis is planned and the study has multiple stages, for simplicity it is assumed that equal number of subjects are enrolled in each stage. For $k=1,\dots ,K$ baskets, let ${n_{ck}}$ and ${n_{tk}}$ denote the control and treatment arm sample sizes for basket k in each stage, respectively. Let Y denote the outcomes, with subscripts c and t denote the control arm and treatment arm. Write $i=1,\dots ,{n_{ck}}$ and ${i^{\prime }}=1,\dots ,{n_{tk}}$ as individual subjects in the control and treatment groups. In addition, the outcomes are assumed to be normally distributed, and the subject level variation is the same across the whole population, denoted by ${\sigma ^{2}}$.

The majority of the literature discussed information borrowing methods in basket trials without considering the possible diversity of patients across the baskets. It is reasonable to speculate that the outcomes of basket trials may depend on the participants’ baseline characteristics, such as gender and age. Also, it is known that randomization is a common feature of clinical trials; however, complete randomization may not be achievable in reality for a variety of reasons. U.S. Food and Drug Administration [40] provides general guidance on which covariates should be included in the modeling of a clinical trial: including the adjustment of stratification factors for randomization and important baseline characteristics that are strong predictors for the outcome. In such cases, adjustments for covariates are necessary, as discussed in [44] and [43]. In this paper, we also model the outcomes by including covariates effects, with the following assumptions and notation.

Assume the outcome depends on r covariates. In the k-th basket, denote ${\tau _{k}}$ as the treatment effect. For the k-th basket, the covariates vectors are ${\boldsymbol{x}_{cki}}$ and ${\boldsymbol{x}_{tk{i^{\prime }}}}$, respectively, for individual i in the control arm and for individual ${i^{\prime }}$ in the treatment arm, where ${\boldsymbol{x}_{cki}}={(1,{x_{cki1}},\dots ,{x_{ckir}})^{\prime }}$ for the control arm and ${\boldsymbol{x}_{tk{i^{\prime }}}}={(1,{x_{tk{i^{\prime }}1}},\dots ,{x_{tk{i^{\prime }}r}})^{\prime }}$ for the treatment arm are $(r+1)\times 1$ vectors. The coefficient vectors ${\boldsymbol{\beta }_{k}}={({\beta _{k0}},{\beta _{k1}},\dots ,{\beta _{kr}})^{\prime }}$ are assumed to be different among baskets. Note the “${\beta _{k0}}$” is interpreted as the control group effect for k-th basket (after adjusting covariates). Denote $\{({\boldsymbol{x}_{ck1}},{y_{ck1}}),\dots ,({\boldsymbol{x}_{ck{n_{ck}}}},{y_{ck{n_{ck}}}})\}$ and $\{({\boldsymbol{x}_{tk1}},{y_{tk1}}),\dots ,({\boldsymbol{x}_{tk{n_{tk}}}},{y_{tk{n_{tk}}}})\}$ for $k=1,\dots ,K$ to be the data observations. The outcomes are assumed to follow If there is no covariate that need to be adjusted, the above formulae will be simplified with ${\boldsymbol{x}_{cki}}=1$, ${\boldsymbol{x}_{tk{i^{\prime }}}}=1$ and the coefficients ${\boldsymbol{\beta }_{k}}={\beta _{k0}}$.

The main objective is to detect whether the treatment is superior to the control in any of the K baskets. This corresponds to testing the following hypotheses: for $k=1,\dots ,K$. The value of δ represents the magnitude of improvement over the control arm needed to declare a clinical benefit of the new treatment [44] and needs to be specified in advance. Implicitly, a positive ${\tau _{k}}$ indicates the drug or treatment is effective.

We are piloting two study designs: a one-stage design without any Interim Analysis (IA) and a two-stage design with one IA. Let “$\mathcal{D}$” denote the available data collected at the time point of the analysis. Let ${\mathcal{D}_{IA}}$ denote all available data at the interim analysis within the two-stage design; and ${\mathcal{D}_{FA}}$ denote all available data at the Final Analysis (FA). For trials with the two-stage design, we adopt similar decision rule proposed by Mehta and Pocock [17]: at the IA, a basket could stop early for futility or efficacy, or enter the “promising zone”: meaning the recruitment continues and it is evaluated in the next stage. In the final stage, a basket could be claimed as futility or efficacy.

The inference is performed using the posterior probability based on the data available at the current stage, i.e. Two cut-off values $0\lt {q_{1}}\lt {q_{2}}\lt 1$ are pre-set for futility stopping and efficacy stopping, respectively. Below describes the decision criteria for the stopping rules:

1. At IA,

2. At FA,

There criteria are illustrated graphically in Figure 1.

There are various methods to determine the cut-off values ${q_{1}}$ and ${q_{2}}$. Usually, they shall imply the percentage level required to certify that the treatment is compellingly improved over the control. In our simulation study, we link ${q_{1}}$ and ${q_{2}}$ by a parameter $0\le \eta \lt 1$: $1-{q_{2}}=\eta (1-{q_{1}})$. η is essentially the ratio between the “effective zone” and the “effective zone” plus “promising zone”, thus controlling alpha-splitting in the two stages. Specially, when $\eta =0$, there is no chance claiming success at IA: only futility rule is applied. For η between 0 and 1, there will be a “promising zone” and both efficacy and futility rules are applied. Further, the cut-off values are determined by allowing the trial to comply with certain statistical constraints: under the scenario of the global null (GN) (i.e. the treatment effect is zero in every basket), controlling Family-Wise Error Rate (FWER) at the final stage to be 10%. The details of setting the rules are given in Section 4.3.

The sample sizes are assumed to be equally assigned in the two stages. The sample sizes of the two stages can be different in principle, and the allocation between stages is related to alpha-slitting. The calculation of overall sample size as well as allocation between stages is not the focus of this article, but may be a topic worth exploring.

A major concern of the basket trial is that the sample size is often too small to achieve a desirable power. This relates to the accuracy of the variance estimates. Under our assumption that the response outcomes share the same population variation, a favorable set up is estimating the population variance by aggregating all the data across individual baskets. Assume conjugate priors for the normal likelihood function to facilitate the posterior derivation and save computation time. Thus in our modeling, data are assumed to be normal with conjugate priors and the population variance is given an Inverse Gamma prior. These assumptions are universal for all the methods applied in this article.

Note that as the assumptions of the basket trial are extended in many aspects, it is currently difficult to find a published study dataset of basket trials that satisfies these assumptions at the same time. We do find one example mentioned by Ouma et al. [23]: a study trial comparing pulse rates after participants performed three kinds of exercises between different dietary interventions. This study is similar to a basket trial with a controlled arm. Two diet types (low-fat and non low-fat) are considered, and low-fat is set as the control. The data source included two sub-studies that can be regarded as two stages. Each sub-study involved 30 participants randomly assigned to three exercises and two diets. Pulse rate measurement is the continuous endpoint of interest.4

We attempted to explore two model designs, the first assuming that the outcome estimates (pulse rate after exercise) are independent of any covariates; the second assessing the change from the baseline and model the outcome with adjusting the baseline values as the covariate. The results are summarized in Tables 11, 12 and 13 for one-stage design, two-stage design with futility and efficacy, and two-stage design with futility only, respectively. Also, note the one-stage design applied all data from the two sub-studies instead of only from the first sub-study.

The performance of the model may depend on the tuning parameters in our simulations, for illustration purposes we simply placed the non-informative prior in this example. The estimates of treatment effects and their 95% Highest Posterior Density (HPD) Interval are summarized in the tables. The cut-off value is firstly determined based on the simulated global null (GN) scenario. The GN scenario was simulated by bootstrapping the control arm data. The cut-off values of ${q_{1}}$ controlled the FWER to be 10% under GN and they are summarized in the result Tables. The posterior probability $Pr({\tau _{k}}\gt 0|\mathcal{D})$ are also summarized to be compared with ${q_{1}}$.

For one-stage design, since all data are included in the analysis, borrowing resulted in the overestimate of the effects in small-effect-baskets such as baskets 1 and 2. In contrast, a two-stage design allows baskets that meet the futility and/or efficacy stopping rule to drop at interim analysis. Thus the two-stage designs helped reducing Type I Error Rate inflation and the estimates of baskets 1 and 2 are not much overestimated. BHM tend to have the largest estimates for baskets 1 and 2, which is also reflected in the high Type I error. In Table 13, the reason that all methods have the same estimates for basket 3 is that the other two baskets are dropped in stage one, and only basket 3 enters the second stage and it is evaluated the same way as SEP. In all designs, MFM tends to clustered baskets 1 and 2 together as they both have small effects. Comparing the methods discussed so far, a similar pattern is observed in our simulation results.

Basket trials study design are innovative for clinical development, which can bring effective treatment to patients faster. To achieve full strength of basket trials, sophisticated statistical analysis methods are needed. In the context of identifying true efficacious treatment for further evaluation, we pilot four Bayesian methods when apply to 3 study designs, and have demonstrated the feasibility of adding a control arm as well as adding covariates to the modeling.

Modifications to CBHM and MFM are proposed. For CBHM, we extend [6] to the continuous outcomes with a few changes to obtain similar performance. For MFM, tuning parameters are introduced into the prior settings to optimize the borrowing effect. Based on our simulation, each of CBHM and MFM performs better than others in some but not all scenarios. In general, MFM has more robust performance, and it has more power while lower inflated FWER than other methods in a wide range of scenarios.

For the one-stage design, the CBHM is an improvement over the BHM in terms of better control of the FWER while sacrificing a small amount of power. Both models surpass the power of the Bayesian Separate Model. For the two-stage design with only a futility rule, CBHM and BHM lose some of their power improvements because their full borrowing strategy creates a tendency to borrow from the futility basket in the second stage, dragging down the efficacy basket estimates. When the early efficacy stopping rule is incorporated in the two-stage design, CBHM and BHM gain improvement because the efficacy stopping rule prevents the truly active baskets from being mixed with the ineffective baskets. Meanwhile, MFM shows robustness in all study designs due to its allowance for local borrowing and deliberate prior parameters setting. The potential downside is the computation time when sample size getting large. For moderate sample size, as what this paper has considered, computation time does not seem to be an issue. It only take a few hours to complete 1000 simulations. Another downside is the lack of standard software for implementation. One needs to derive the complicated formulas to run model for different applications.

In our research, many assumptions were set at the beginning in order to test the modeling performance and study design. However, many of the assumptions can be either removed or made more flexible. First, all methods may assume that the baskets have different within basket variances rather than a unified variance. To implement this, update the Gibbs sampling algorithm for BHM, CBHM and MFM, e.g. update ${\sigma _{k}^{2}}$ instead of ${\sigma ^{2}}$ in (3.15) and (3.21). Second, the sample size can be different between the control and treatment arms, and between stages. In addition, the sample size determination is not our focus thus not mentioned in this article. Moreover, we arbitrarily set $\delta =0$ for all stages; and set the parameter $\eta =1/2$ that links the two cut-off values of futility and efficacy stopping rules. These settings may affect the alpha-split in two-stage trials. Many different considerations of these settings appear in the literature. All of the aforementioned may not be the main research focus in this work, but may be potential extensions for future work.

Another important extension of the work is the consideration of other types of endpoints, for example, count data and time-to-event data. Some of the models that can be considered are the Poisson regression model and the proportional hazards model. In a similar way to a continuous endpoint, a Bayesian hierarchical structure can be added to the treatment effect part of the model. When non-conjugacy modeling is considered, other sampling algorithms can be explored for both CBHM and MFM. Some references can be found in articles by Miller and Harrison [18, 19], and Neal [20].