Longitudinal studies are common in clinical studies and involve multiple measurements of the subject over time. For example, subjects in a clinical trial are recruited and randomized into treatment groups and responses from each patient are observed multiple times over a user-selected period of time. There is a huge literature on analyzing longitudinal models using different methods but design issues for such studies lagged. What seems to be generally known is that having excessive time points does not necessarily improve the quality of the statistical inference and having too few observations per patient to facilitate calculation may not meet the scientific requirements in the study.

Given a statistical model and a design criterion, design issues for a longitudinal study concern the optimal number of time points to observe the outcomes, where the time points are over the study period and the number of replicates at each time point. For making statistical inference as accurately as possible, these decisions have to be made judiciously and at minimum cost. Analytical approaches are difficult because they invariably involve number-theory to solve the optimization problem. Even when it is possible to derive the results mathematically, they can be limiting because they are valid to a single statistical model and does not apply to even a slightly changed model. Software and efficient computational methods are therefore useful for answering such design questions.

We propose using nature-inspired metaheuristic algorithms to tackle these challenging questions. This class of algorithms is commonly used in engineering and computer science to tackle hard to solve optimization problems but surprisingly under-used in mainstream statistical research. We focus on particle swarm optimization (PSO), which is a popular member of this class of algorithms and apply it to design longitudinal studies with different mean functions and various correlation structures. Such algorithms typically have variants, which are enhancements motivated by different desires to improve certain aspects of its performance. We also compare results from PSO with its various enhancements and determine whether the latter provide markedly superior results than the results from the original PSO.

To fix the idea, we discuss the proposed methodology for the 2-parameter nonlinear Michaelis-Menten model which is widely used to study enzyme-substrate dose relationship in kinetic biological systems. Its simplicity and usefulness make it one of the most widely used models across many disciplines, such as, in agriculture [37], biochemistry [19], biology [4], environmental study [36]. [35] notes that the Michaelis-Menten model is a special case of the 3-parameter logistic model, which is more widely used in practice.

In its simplest form, the Michaelis-Menten model for a chemical reaction is where υ is the observed velocity of the reaction when the substrate concentration applied is $t\in T$, where S is the user-selected range from which concentration is selected. The mean response is a nonlinear function $\eta (\theta ,t)$ with 2 parameters a and b. The parameter b is called the Michaelis-Menten constant, which controls the rate of the reaction and so between the two parameters, it is the more biologically meaningful parameter. The maximum velocity attainable is a, which is reached when the substrate concentration is increased without bound. Errors in this model are assumed to be independent with mean 0 and constant variance ${\sigma ^{2}}$.

This model can also be used to study the growth curves of animals or subjects by taking repeated measurements on the subjects [17]. Suppose that there are n subjects, and for each subject, m repeated observations are allowed over a period of time. [7] addressed design issues when the responses are correlated using the following model where the error terms ${\epsilon _{i,j}}$ are normally distributed, each with mean 0 and constant variance and ${\epsilon _{i,j}}$ and ${\epsilon _{{i^{\prime }},{j^{\prime }}}}$ are independent if $i\ne {i^{\prime }}$, and each pair of ${\epsilon _{i,j}}$ and ${\epsilon _{i,{j^{\prime }}}}$ has the correlation coefficient ${\lambda ^{\mid {t_{i,j}}-{t_{i,{j^{\prime }}}}\mid }}$, $j,{j^{\prime }}=1,\dots ,m$. Here $\lambda \in (0,1)$ and ${t_{i,{j^{\prime }}}}\in S$. Given n and an optimality criterion, design issues for the model concern how to optimally select m, the number of time points and the values of the s to observe outcomes so that the criterion is optimized. A typical criterion is to estimate the parameters a and b or some function thereof as accurately as possible.

Throughout, we assume the following setup. We are given a total of N observations for the study and a regression model defined on a given design space S. Given an optimality criterion, finding an optimal exact design means finding the optimal value of the criterion by choosing the optimal number of points k to observe the response, where the time points and the number of replicates ${n_{i}}$ at each time point ${t_{i}}$ so that ${n_{1}}+{n_{2}}+\cdots +{n_{k}}=N$. A complicating feature of such an optimization problem is that there is neither a general theory for finding an optimal exact design nor a theoretical tool for confirming the optimality of an exact design. Consequently, optimal exact designs are rarely studied in the literature, especially for nonlinear models. As an early example, [3] found D-optimal exact designs numerically for homoscedastic polynomial models of low degrees.

Our goal is to investigate the effectiveness of using a nature-inspired metaheuristic algorithms, such as particle swarm optimization (PSO), for finding D-optimal exact designs for the Michaelis-Menten model and related models with correlated errors, c-optimal designs for estimating a function of the parameters in a growth curve model, and a maximin optimal exact design for optimizing efficiencies in a HIV dynamic model. We show our results agree with the theoretical designs in [7], which are only available for small values of m and n and able to find the optimum in all problems quite quickly. However, there is no general theory for finding optimal exact designs or confirming optimality of an exact design, which may, in part, explain why exact designs are much less studied than approximate designs, where there is a general framework and general methods for finding and conforming optimality of an approximate design when the criterion is convex functional.

The rest of the article is organized as follows. Section 2 briefly reviews particle swarm optimization. Section 3 applies PSO to search for locally D-optimal exact designs for the Michaelis-Menten model with various correlation structures. The Michaelis-Menten model is a nonlinear model, and, as we explain below, its information matrix and hence the design criterion depends on unknown model parameters that we wish to estimate. The simplest design strategy is to use results from a pilot study or similar studies to have a good guess of the true parameters that we wish to estimate. These nominal values, which may be also available from experts, are then substituted into the design criterion and the problem then involves optimizing the design parameters only. We next discuss a few PSO enhancements in Section 3.2 and compare their performance for finding locally D-optimal designs for the Michaelis-Menten model with various error structures. Additionally, we apply PSO to find locally c-optimal designs for estimating one or more functions for a generalized version of the Michaelis-Menten model for studying growth rates in Section 4. Section 5 considers an alternative to locally optimal design and constructs maximin locally exact designs for a longitudinal study to estimate parameters for a HIV dynamic model. Section 6 concludes with a short discussion.

Recently a class of algorithms called nature-inspired metaheuristic algorithms has proved very popular in the optimization literature. [31, 32] provided reasons for the rapid rise and interest in these algorithms. Early users of such algorithms to find optimal exact designs for linear models include [12] who used an annealing algorithm to search for optimal designs for linear regression models, and [20] who used a genetic algorithm to construct exact D-optimal designs. Of particular note is the particle swarm optimization (PSO) introduced by [8] for tackling optimization problems. PSO is increasingly used across disciplines to solve hard optimization problems. PSO is essentially assumptions free and it searches in a simple and effective way. For example, unlike many algorithms, PSO does not require the objective function to be differentiable or convex and can solve non-convex high-dimensional optimization problems.

PSO is a metaheuristic optimization algorithm inspired from the way animals, such as birds and fishes, search for food. The birds fly continuously in the sky to look for food on the ground. Each has its own perception where the food is (local optimum) but it communicates with the rest and collectively the flock decides where the food is (global optimum). Accordingly, each bird flies toward the global optimum in the direction of the local optimum (not giving up completely where it thinks the food is). Birds are referred as particles and each bird represents a candidate solution to the optimization problem. Velocities and locations of each bird are adjusted at each iteration and if and when the flock converges, the perceived global optimum is found. In order to efficiently identify the optimal points, we initiate a flock of birds in the pre-defined search space. Let $X(k)$ be the locations of particles at the k-th iteration. Define ${t_{L}}(k-1)$ to be the locations with the best objective function values discovered by each particle before the k-th iteration, and ${t_{G}}(k-1)$ to be the locations of the best value found by the whole swarm before the k-th iteration. At the k-th iteration, the particles are updated by and where $V(k)$ is the velocity of the particle. There are several parameters in (2.2). The inertia weight represents how active the birds are and is denoted by w. This parameter may be chosen to be a positive constant but more typically its value changes over iteration and eventually decrease to 0. The parameters ${c_{1}}$ and ${c_{2}}$ are two positive constants which are recommended to be 2, and ${R_{1}}$ and ${R_{2}}$ are two random vectors whose components are independently drawn from the uniform variate on $[0,1]$. In practice, the number of iterations and the swarm size are the most influential parameters in PSO. Large swarm size gives a better vision on the search area such that PSO could achieve the global optimum with a higher chance, and the more iterations allows the particles having search experience due to random perturbation. More details on PSO and the related metaheuristic optimization algorithms are available in [34].

For an individual patient with $n=1$ and $N=m$, we determine the optimal N time points to observe responses from the patient. We allow replications, i.e. the N time points need not be distinct. To search for a N-point optimal exact design, we search for an optimal $N\times 1$ vector, $({t_{1}},{t_{2}},\dots ,{t_{N}})$ and allow some of the ${t_{i}}$’s may be the same. Here N is user-specified and the objective function is the design criterion $\Phi (\cdot )$, which may not be convex function. A key challenge is the dimension of the problem when N is large. When there are multiple optimal solutions, and we only report one of them. Hence at the beginning of the PSO, we randomly generate initial particles (designs) on the design space. Then at each iteration, we update the particles (designs) based on the (2.1) and (2.2). We also watch out for those that flew outside the search boundaries and when this happens, we need to adjust those particles to make sure they are in the design space properly. The hope is that after a number of iterations, the particles will converge to a point and this point is supposedly the global best solution or the optimal design we are after. Algorithm 1 below is a pseudo code of the PSO algorithm for finding an optimal design.

In the next section and beyond, we apply PSO-based algorithms to find various types of optimal exact designs for models with uncorrelated errors and errors with various correlation structures and compare the designs. Throughout, the hardware we used is a PC with 3.50 GHz Intel(R) Core(TM) i7-4770K CPU.

Suppose in a longitudinal study, we have n subjects and each is observed at m time points ${t_{ij}},j=1,\dots ,m$ from a time interval T. We assume T, m and n are given. The normalized information matrix is proportional to where $N=nm$, ${\theta _{0}^{\top }}=(a,b,{\sigma ^{2}},\lambda )$ and ${f_{i}}$ is the $m\times 2$ matrix with the j-th row containing the derivative of ${f_{i}}$, which is

The $m\times m$ correlation matrix of the m responses from the ith subject is ${\Sigma _{i}}$ and its $(j,k)$-th element is Thus, given a fixed parameters ${\theta _{0}}={(a,b,{\sigma ^{2}},\lambda )^{\top }}$, the locally D-optimal design for estimating the model parameters as accurately as possible is the one that maximizes over all designs ξ on T. Following the argument in [7], we may, without loss of generality, set $T=[0,1]$. For the Michaelis-Menten model, locally D-optimal designs do not depend on a and ${\sigma ^{2}}$ and so, we choose $a={\sigma ^{2}}=1$. We also assume that the experimental conditions for each subject are the same and each subject is observed at the same set of time points, ${t_{1j}}={t_{2j}}=\cdots ={t_{nj}}$, for $j=1,\dots ,m$. Thus, to search for the locally D-optimal exact design via PSO, we can set $n=1$, i.e. $N=1\times m=m$, and the target design is represented as an $m\times 1$ vector.

We first consider the case when we have 2 observations from a single subject and the design questions are which two time points to take observations and how the spread out the observations between the two points. [7] theoretically identified that the locally D-optimal exact design is supported at two points, u and 1, when $b\ge \frac{1}{3}$, and the design point u solves the equation The solution has no closed form when $b\lt 1/3$. When there are 3 or 4 observations to be taken from each subject, locally D-optimal exact designs cannot be described analytically and only numerical results were provided. The numerical approach they used is to check all possible designs using a pre-specified grid.

We next use Algorithm 1 to systematically find the locally D-optimal exact designs. We used 256 particles and set the maximum number of iterations equals to 500. The PSO parameters ${c_{1}}$ and ${c_{2}}$ were set to their default values with both equal to 2. For the inertia weight, w, we started from 0.95 and linearly decreased it to 0.4 in the first 350 iterations and then fixed w as 0.4 for the remaining iterations. We implemented the algorithm and found PSO-generated D-optimal designs for several values of b and λ, i.e. $b=0.2,0.7,1.2,1.7,2.2,2.7$ and $\lambda =0.1,0.5,0.9$. We first consider the cases when the numbers of observations per subject were $m=2,3,4$ and a direct application of PSO shows the generated designs coincided with the analytical designs in [7] when $m=2$ and $b\ge 1/3$. The PSO-generated designs when we have 3 or 4 observations per subject also agree with those numerically found optimal designs from [7]. For space consideration, we do not display them. We next use Algorithm 1 to search for D-optimal designs when there are more observations per subject. Table 1 displays the PSO-generated optimal exact designs for 5 and 6 observations per subject. Unlike the global numerical search approach in [7], we are not constrained by the grid size imposed on the experimental region, S, and PSO can still identify the best designs efficiently. The CPU time required by our Algorithm 1 to find the optimal design is short. On average, our MATLAB codes take around 4.11, 5.54, 6.72, 7.83 and 8.71 seconds to find the optimal designs when there are 2, 3, 4, 5 and 6 observations, respectively.

In summary, Algorithm 1 seems efficient for finding locally D-optimal exact designs for the Micahelis-Menten model. Tables 2 to 4 show PSO-generated designs have more design points as the value of m is increased. To conserve space, we consider particular cases of ${(\lambda ,b)^{\top }}$ when λ is fixed at 0.1, 0.5 and 0.9, and $b=0.5,1.0,1.5,2.0$ and 2.5.

A growth curve describes the change of an outcome over time. To estimate a growth curve model, we take multiple observations at different time points from the same subject and so the observed responses are correlated.

A generalized version of the Michaelis-Menten model to study the biomass W of an animal over time was introduced by [17]. At time t, the model is as follows: where ${W_{0}}$ and ${W_{f}}$ are the zero- and infinite-time values of the biomass, respectively. If $K\gt 0$, it is the time when half-maximal growth is achieved. When $h=1$ and ${W_{0}}=0$, model (4.1) reduces to the usual Michaelis-Menten model.

We consider the data set from [17] consisting of 17 weight (kg) records of one castrated male Percheron horse from birth to 220 weeks of age. Its growth pattern has been described by the generalized Michaelis-Menten model with nominal values ${\theta _{0}}={({W_{0}},{W_{f}},K,h)^{\top }}={(85.50,731.00,56.70,1.39)^{\top }}$ in [17]. Using ${\theta _{0}}$, they found a locally D-optimal exact design, ${\xi _{D}^{\ast }}$ for estimating parameters in (4.1) by maximizing the determinant of the Fisher information matrix. For model (4.1), the elements in the covariance matrix are ${\Sigma _{ij}}={\sigma ^{2}}\exp \{(-\lambda |{t_{i}}-{t_{j}}|\}$ and

Human immunodeficiency viruses (HIV) are a subgroup of retrovirus that can cause infection, resulting in progressive failure of the immune system. The presence of viruses is usually detected when the CD4 cell counts are low. Monitoring the cell counts continuously is key to understanding disease progression. Various statistical models have been proposed to model CD4 cell count longitudinally. A common model is to assume a fixed effect model defined on a time line in hours over a pre-specified time interval, say, $t\in T=[0,6\frac{11}{12}]=[0,6.917]$,

The above time interval is taken from Table 1 in [13] that describes HIV RNA copies per milliliter plasma and time is measured since treatment initiation and the predictor, t, is time after pharmacologic delay. Frequently interest is in estimating one or more of the parameters in the model parameters since not every parameter has a meaningful biological interpretation. When all parameters are interesting, the D-criterion discussed previously is an appropriate design objective; otherwise, if only a subset of the parameters is interesting, the design criterion is c-optimality. In the above model, we are interested to find an optimal design to estimate ${\theta ^{T}}=(\log {V_{0}},\log c,\log \delta )$ and optimal designs to best estimate each of its components. Additionally, we also find a maximin optimal design goal that maximizes the lowest efficiency among the three efficiencies for estimating each of the three parameters.

For such studies, there is usually a standard protocol recommended by physicians for implementation. The protocol specifies how many times and when to sample blood to determine cd4 counts and for other laboratory tests. However, the protocol is usually not based on statistical considerations. We now apply PSO to find various optimal designs and ascertain how efficient is the protocol design under various scenarios, including situations where there are multiple objectives and there are different emphases in each of the study objectives.

We compute various optimal exact designs and evaluate the efficiencies of the recommended or protocol design or baseline design for such a study. For a given number of points, uniform designs have them evenly distributed on the time interval T. For example, such a design with 8 observations is ${\xi _{8}}=\{0.000,0.917,1.917,2.917,3.917,4.917,5.917,6.917\}$. A direct calculation shows its Fisher information matrix is proportional to where the ${(ij)^{th}}$ element of the matrix F is ${F_{ij}}(t)=\partial \eta ({\theta _{i}},t)/\partial {\theta _{j}}$ and $\eta (\theta ,t)$ is the mean response at time $t\in T$. For estimating all parameters in the model, D-optimality is commonly used and is defined by the determinant of $M(\xi ,\theta )$. Because the information matrix contains unknown parameters, we replace θ by its nominal values before we maximize the determinant using PSO. Using nominal values ${\theta ^{\top }}={(\log {V_{0}},\log c,\log \delta )^{\top }}=(11.0,1.1,-1.0)$, the resulting PSO-generated locally D-optimal design with 8 observations is ${\xi _{D}}=\left\{0.000,0.000,0.000,2.083,2.083,6.917,6.917,6.917\right\}$. This implies that the implemented design requires 3 replicates at 0, 2 replicates at 2.08 and 3 replicates at 6.92.

To estimate $\log c$ or $\log \delta $, we find an exact design that minimizes the asymptotic variance of the estimated function. Setting ${c^{\top }}=(0,1,0)$ to estimate $\log c$, ${c^{\top }}=(0,0,1)$ to estimate $\log \delta $ in (4.2) and using the given nominal values for the model parameters, the PSO-generated designs for estimating $\log c$- and $\log \delta $ are and

These designs can be implemented in practice as discussed for the PSO-generated 8-point D-optimal exact design. When there are multiple objectives in the study, one considers them simultaneously at the design stage. For example, when we wish to estimate θ, $\log c$ and $\log \delta $, we want a design that provides high efficiencies under all three criteria. One way is to use a maximin optimality criterion and find an exact design ${\xi _{MM}}$ that maximizes the minimal efficiency across the three criteria:

The first of the three terms in the outer curly brackets represents the D-efficiency (D-eff) of the design ξ and the last two terms represent the c-efficiencies (c-eff) of the design ξ of estimating $\log c$ and $\log \delta $. In practice, designs with high efficiencies are sought. If the efficiency of a design ξ is 0.5 or $50\% $, this means that the design ξ needs to be replicated twice to perform as well as the optimal design.

The search for the locally maximin optimal design is a two-step approach. First, we identify the three locally D- and c-optimal designs required to compute the efficiency values in (5.3). Second, we use PSO to search for the maximin optimal design that maximizes (5.3). In particular, the criterion is the minimal value among three efficiencies and thus at each iteration, PSO needs to find the optimal design, compute these three relative efficiencies first and then identify the minimal value as the criterion value.

For the given set of nominal values, Algorithm 2 produced the 8-point PSO-generated maximin design: ${\xi _{MM}}=\left\{0.000,0.000,1.847,1.847,1.847,1.849,6.917,6.917\right\}$, implying that the implemented design requires 2 replicates at 0, 4 replicates at 1.85 and 2 replicates at 6.92.

Table 15 displays the various relative efficiencies among the protocol design, ${\xi _{8}}$, the locally D-optimal design, ${\xi _{D}}$, the locally c-optimal designs, ${\xi _{\log c}}$ and ${\xi _{\log \delta }}$ and the locally maximin optimal design, ${\xi _{MM}}$.

The protocol design ${\xi _{8}}$ has low c-efficiencies for estimating $\log c$ and $\log \delta $, averaging about $45\% $ and substantially higher D-efficiency, about $72\% $ for estimating all parameters in the model. The locally D-optimal design has at least 67% c-efficiency, which is acceptable for estimating the individual parameters $\log c$ and $\log \delta $. When one of the two model parameters log c or log δ is of interest, the c-efficiency of the optimal design for estimating the other parameter is not high; Table 15 shows they are $48.33\% $ and $54.25\% $, suggesting that they are not robust to mis-specification in the optimality criterion. This is in contrast to the maximin optimal design, which has at least $81.31\% $-efficiency across all three criteria. This suggests that when we are unsure at the onset which of the parameters are more interesting to estimate, it may be desirable to implement a maximin optimal design.

Optimal exact designs are rarely reported in the literature because they are harder to find and study analytically even when the sample size is small. It is essentially a number-theoretical problem, where invariably, solutions, have to be derived for each problem and frequently, for different values of N as well. Further, the derivation of the analytical optimal design for a model is inapplicable to a slightly changed model, and the few algorithms for finding optimal exact designs are restrictive and usually for relatively simple linear models only; see for example, [3].

This paper investigates PSO’s capability to generate a variety of exact designs for various biomedical nonlinear models when errors in a nonlinear model may be correlated and there is one or more design criteria in the study. We implemented PSO using R codes to find them; for example, Algorithm 1 generates the locally D-optimal designs for the Michaelis-Menten model in this paper.

In all cases, PSO was able to successfully and efficiently identify the best designs and they agreed with the theoretical exact D-optimal designs when the latter are available for simpler setups. Unfortunately, there are no theoretical checks whether the PSO-generated optimal exact designs are truly optimal. One way to assess its validity is to compare the optimal exact design with the corresponding optimal approximate design, which can be found using standard algorithms and confirmed theoretically via an equivalence theorem. To this end, we also apply PSO to find the corresponding optimal approximate designs for models studied in the paper and we were able to confirm that both the generated PSO exact designs and the optimal approximate designs are close. For example, for the design problem discussed in the last section, we found the relative D-efficiency is 98.28% and the c-efficiencies for estimating $\log c$ and $\log \delta $ are $99.78\% $, and $94.29\% $, respectively.

Because PSO is a general optimization technique and requires little or no assumption on the problem for it to be applicable, we expect PSO should also perform well in finding other optimal exact designs for different types of models with various correlation structures, including high dimensional models. We also encourage further exploration and application of PSO to solve general optimization problems in statistics.