The Supplementary Material to “General Additive Network Effect Models” contains additional simulation results for Section
In the interest of business innovation, social network companies often carry out experiments to test product changes and new ideas. In such experiments, users are typically assigned to one of two experimental conditions with some outcome of interest observed and compared. In this setting, the outcome of one user may be influenced by not only the condition to which they are assigned but also the conditions of other users via their network connections. This challenges classical experimental design and analysis methodologies and requires specialized methods. We introduce the general additive network effect (GANE) model, which encompasses many existing outcome models in the literature under a unified model-based framework. The model is both interpretable and flexible in modeling the treatment effect as well as the network influence. We show that (quasi) maximum likelihood estimators are consistent and asymptotically normal for a family of model specifications. Quantities of interest such as the global treatment effect are defined and expressed as functions of the GANE model parameters, and hence inference can be carried out using likelihood theory. We further propose the “power-degree” (POW-DEG) specification of the GANE model. The performance of POW-DEG and other specifications of the GANE model are investigated via simulations. Under model misspecification, the POW-DEG specification appears to work well. Finally, we study the characteristics of good experimental designs for the POW-DEG specification. We find that graph-cluster randomization and balanced designs are not necessarily optimal for precise estimation of the global treatment effect, indicating the need for alternative design strategies.
Network data encode not only information about individuals but also the connections among them. Such data exist in many settings: students who are friends with one another [
Network of Facebook connections among 770 Caltech Facebook users. The network was plotted using the
The dependency structure in network data poses distinctive challenges and opportunities and thus motivates specialized theory and methodology. In this paper, we focus on network experimentation, a topic that has gained recent interest given the widespread experimentation happening in social network companies like LinkedIn, Facebook, and Twitter [
Current literature on network experimentation generally focuses on the setting where the experiment is conducted on a given network of a fixed number of individuals. Connections among individuals on the network are further assumed to be known and unchanged over the duration of the experiment. In addition, the experiment usually involves only two experimental conditions: the proposed new condition (i.e., the
There are two stages in running an experiment: designing the experiment and analyzing the results. In the design stage, the experimenter determines the treatment assignment for the experimental units. Then in the analysis stage, data collected from the experiment are used to estimate the effect of the treatment on the outcome of interest. A popular estimand in the literature is the
Classical experimental design and analysis methodologies treat individuals (or experimental units) as independent entities, in the sense that the potential outcome of each unit is assumed to be independent of the experimental conditions the
There are two major classes of approaches to the design and analysis of experiments on networks. The first base their methods on the exposure framework [
To achieve a precise estimator of the global treatment effect, each exposure group needs to contain a large number of units. In other words, the exposure-based analysis strategy requires that experimental units are mostly surrounded by neighbors assigned to the same experimental conditions as themselves. This motivates a popular design strategy in the network experimentation literature based on
The second class of approaches is based on assuming a statistical model for the experimental outcomes. There is a rich literature on network regression models for inference and prediction problems [
In the model-based framework, once the model is formalized, experimenters can select design criteria to reduce, for example, the variance of the model parameters and/or functions of them. Designs that optimize the posed design criteria can be found using exhaustive search [
Of the two classes of approaches, the exposure framework seems more popular in the network experimentation literature thanks to its focus on the global treatment effect. When the main goal of the experiment is to accurately estimate the global treatment effect, the exposure framework only requires the experimenters to define the treatment and control exposures. For instance, in the Gui et al. [
On the contrary, if the generating model for the experimental outcome is correctly specified, the model-based framework will have several advantages over the exposure framework. First, data from all units participating in the experiment can be utilized when fitting the model in the analysis stage. Second, additional inference beyond the global treatment effect can be achieved via model parameters and functions of them. Finally, design selection can be calibrated according to the experimenters’ specific interests, which may sometimes be something other than the global treatment effect.
In this paper, we attempt to unify many of the model-based approaches by introducing the general additive network effect (GANE) model, in which the experimental outcome of a unit is modeled as an additive function of the effect of its own treatment assignment and the effects that come from other treated or controlled nodes in the network. The model not only encompasses (as special cases) many network experiment models from the literature, but it also flexibly extends the manner in which network effects are modeled. In particular, the network influence from the treatment and control groups can be modeled differently in terms of both size and functional form. Moreover, the model is specified such that its parameters are interpretable. We show that (quasi) maximum likelihood estimation is possible for a family of model specifications. The resulting (quasi) maximum likelihood estimators are then proven to be consistent and asymptotically normal, which facilitates familiar likelihood-based inference.
Existing work in the model-based direction concentrates on inference for the model parameters themselves, however, in many cases, the global treatment effect is the main quantity of interest. To expand the utility of the model-based framework, we illustrate how quantities such as the global treatment effect can be expressed as functions of the GANE model parameters, which permits inference via the delta method [
We further propose a particular specification of the GANE model, the power-degree (POW-DEG) specification, where the network effects are modeled as a nonlinear function of the number of neighbors assigned to treatment or control. Using the power as an additional parameter, the POW-DEG model specification gains additional flexibility in capturing the network effect compared to the linear counterpart that has been proposed in the literature [
As with any model-based approach, the GANE model framework faces the challenge of model selection. When the GANE model is estimated via maximum likelihood, experimenters can use popular model selection criteria such as AIC, BIC, etc. to choose the model specification that best fits the data. However, in the design phase, before data have been collected, experimenters must use their domain knowledge to select a suitable model specification, with which they can build design criteria and select a design. In the absence of information concerning which model specification is appropriate, we suggest the use of the POW-DEG specification because of its flexibility and good performance in terms of global treatment effect estimation and inference. Nevertheless, model selection for both the design and analysis of experiments on networks remains an open area for future study.
The rest of the paper is structured as follows. In Section
Suppose that the experiment is on a network of
Parker et al. [
However, on a network, the outcome of a unit may not only be influenced by the
The GANE model has a structure similar to that of the LNE model (
The GANE model encompasses several existing models in the literature, a few of which are identified below.
The above are existing models that have been proposed and discussed in the literature. We next propose a new specification under the GANE framework.
When the number of neighbors is low, for example, in agricultural settings where units are plots placed on a lattice, it may be reasonable to model the network effects homogeneously as in the LNE specification (
Which estimation method is appropriate for fitting the GANE model depends on the specification of the functions
As discussed in Section
Each of these quantities can be expressed as functions of the model parameters
When the GANE model can be estimated using maximum likelihood estimation, since DTE, ITE, and GTE are functions of the model parameters, Hypotheses
To design an A/B test, the experimenter decides which units are assigned to the treatment group and which units are assigned to the control group. This corresponds to specifying the treatment assignment vector
Under the GANE framework, experimenters can define design criteria related to the variance of the parameters’ estimators. For example, D-optimality [
With the design criterion determined, experimenters can find good designs using methods such as exchange algorithms [
We have discussed in Section
In order to obtain the likelihood of the outcome vector
To perform estimation, we consider the matrix form of Model (
With
To find the maximum likelihood estimates, we take the first order derivatives with respect to
Note that in (
Here, we study the behavior of the maximum likelihood estimators as the network size increases to infinity. We use the subscript
The proof of Theorem
With the asymptotic normality result, inference for the parameters can be performed accordingly. The inference for the causal quantities given in Section
In this section, we use simulation to study the properties of different specifications of the GANE model. Specifically, we study our proposed POW-DEG specification (
Number of nodes and edges of the networks used in the simulations.
Networks | # of nodes |
# of edges |
Caltech network | 770 | 16,656 |
UMich network | 3,749 | 81,903 |
We first investigate the asymptotic properties of the maximum likelihood estimates derived in Section
First, we investigate the results for the POW-DEG specification (
The distribution of parameter estimates of the POW-DEG specification on the Caltech Facebook network with
The distribution of the parameter and GTE estimates for the POW-DEG specification (
The variances of the estimates (left axes, lines) and coverage rates (right axes, bars) of POW-DEG specification on the Caltech Facebook network with
The coverage of 95% asymptotic confidence intervals and variances of the parameter estimates are given in Figure
(upper) The distribution of parameter estimates of the HOM specification with
We conducted a similar simulation study on the HOM specification (
As discussed in Section
We study the characteristics of these tests via simulation. Again, as the design
As expected, the rejection rates for each test increase as the respective parameter values depart from their null values. Moreover, tests for
Parameters for the simulation in Section
Specification | ||||||||
SUTVA | 0 | 2 | 0 | 0 | 0 | 0 | 1 | |
LNE | 0 | 1 | 0 | 0 | 0.1231 | 0.1 | 1 | |
POW-DEG | 0 | 1 | 0 | 0 | 0.2691 | 0.1 | 0.5 | 1 |
LAG | 0 | 1 | 0.008492 | 0.001 | 0 | 0 | 0.9977 | |
HOM | 0 | 1 | 0.1 | 0.1 | 0.01728 | 0 | 0.9999 |
Rejection rates of hypothesis tests for POW-DEG specification on the Caltech Facebook network with varying parameters.
We conducted similar simulations for the HOM specification (
The simulations in Sections
In this simulation, on the Caltech Facebook network, outcomes are generated 1,000 times for each of the listed model specifications. The data are then fitted using each of the five model specifications. We use global treatment effect (GTE) estimation and its inference results to compare performance among specifications because the GTE is generally of primary interest. To make the comparison fair, parameters for each model specification are chosen such that the true global treatment effect (GTE) is fixed at 2.0 and the average outcome variance is 1.0 in all data-generating scenarios. The exact parameter values for each specification are provided in Table
Results of the simulation are plotted as heatmaps in Figure
Model misspecification simulation results.
The top right panel shows the variances of the GTE estimates where white represents low variances and dark green represents high variances. We can see that the SUTVA and HOM (
The bottom left panel shows the coverage rate of 95% confidence intervals for the GTE constructed by each estimating model. The results show that LNE (
Finally, on the bottom right, the model selection results by AIC are presented. Green represents high selection rates while white represents low selection rates. AIC works well as it selects the correct model specification most of the time, which is shown by the green diagonal. This supports the use of likelihood-based model selection criteria such as AIC for the GANE framework. Furthermore, it can be seen that POW-DEG specification (
As illustrated in Figure
Given the suggestion for the general use of the POW-DEG specification (
As GTE is the primary focus for many experiments on networks, in this simulation we use the variance of the estimated global treatment effect
For each parameter combination, the best and worst 500 (i.e., 5%) designs with respect to
Characteristics of “best” 5%, randomly selected, and “worst” 5% designs with respect to the variance of estimated global treatment effect on the Caltech Facebook network.
Figure
Table
Efficiency in terms of
Designs | 0.5 | 0.75 | 1.00 | 1.75 | |
1.7521 | 2.3841 | 5.5776 | 23.7801 | ||
Best 500 designs | 1.3875 | 2.0251 | 5.1557 | 23.2936 | |
1.6177 | 2.3152 | 5.5369 | 23.7562 | ||
0.2469 | 0.16627 | 0.0807 | 0.0486 | ||
Worst 500 designs | 0.4772 | 0.2032 | 0.0792 | 0.04791 | |
0.2291 | 0.1588 | 0.0797 | 0.0484 |
We introduce the general additive network effect model for network A/B tests, which unifies many existing models in the literature and enhances the modeling flexibility. We further bridge the model-based framework and the exposure framework by defining causal quantities of interest: the global treatment effect, the direct treatment effect, and the indirect treatment effect as functions of the model parameters. Inference for all three quantities may be carried out via the maximum likelihood framework.
Although the model is studied under the A/B testing setting where there are just two experimental conditions (treatment and control) and under the normal independent error assumptions, the GANE model framework can be extended for use in other settings. First, by expanding the model equation, the GANE model can be used to analyze experiments with more than two experimental conditions. Second, by introducing link functions and other distributional and functional assumptions, the framework can be extended to deal with non-normal distributions and discrete outcomes in manners similar to generalized linear models.
Despite the GANE framework’s wide modeling possibilities, we specifically propose the POW-DEG specification (
The design simulation in Section
We use the following two lemmas.
From the two lemmas, if we have
In order to achieve the asymptotic results in Theorem
The true parameters
The elements of
The matrix
The weight matrices
For each
Assumptions
Note that we cannot use the usual central limit theorems to derive the asymptotic behavior of Model (
Finally, Assumption
To prove consistency, we use the following lemma.
From the reduced form (
Therefore,
Now, to prove the asymptotic normality, we apply the mean-value theorem on the first order derivative of
Therefore, we can apply the above theorem to