Consider the four-platform experiment involving Android, iOS, Windows, and macOS discussed above. Denote the six two-level design factors by 1 , … , 6 on the four platforms denoted by P 1 , … , P 4 . The complete design d of the experiment consists of four sub designs, d 1 , … , d 4 , with d j associated with P j . To quantify the difference among the platforms, let S denote a categorical factor, called the slice factor, with four levels. The jth level of S is associated with P j .

We consider the following properties from [8] to guide the construction of our design:

As a result of Property 1, each sub design d j estimates the effects of design factors on platform P j , and according to effect hierarchy [10, p. 168], the focus of estimation is on the lower-order effects – main effects and two-way interactions. Property 2 suggests that the complete design d focuses on the estimation of the slice factor S and its two-way interactions with the design factors. This requires a different ordering of effects than the effect hierarchy for the complete design d in which S is more likely to be important than the main effects of the design factors, and two-way interaction effects of S with the design factors are more likely to be important than the two-way interaction effects of the design factors. [8] proposed the sliced effect hierarchy for the complete design d in order to accommodate Property 2. To formally define this ordering of effects for the design d in our experiment, let E I be the set of all effects that exclude the slice factor S and E S be the set of all effects that include the slice factor S. [8] defined the sliced effect hierarchy as follows:

In this experiment, the slice factor differs from the design factors in two ways. First, our four-platform experiment aims to detect what level of the design factors should be chosen for each platform and is not trying to select between platforms. Second, according to the sliced effect hierarchy, the importance of the effects related to the slice factor is higher than the importance of the same-order effects of the design factors. A design of the experiment should distinguish between the slice factor effects and the effects of the design factors.

We wanted to use the sliced factorial designs in [8] for our experiment. In a sliced factorial design, each sub design d j follows the effect hierarchy and the complete design d follows the sliced effect hierarchy. Unfortunately, [8] only constructed such designs for two platforms. Since our problem consists of four platforms, we cannot use that method directly. Below we discuss a solution by extending the method in [8] to accommodate our four platforms: Android, iOS, Windows, and macOS.

Our solution generates a design d with 2 6 + 2 − p runs for our experiment, which is a ( 1 2 ) p fraction of a 2 6 + 2 factorial design. First, we describe the construction of a full factorial design d for the experiment. Consider a saturated 2 N − 1 design with N = 2 8 runs. We represent the N − 1 columns of this design by eight independent columns denoted by 1 , … , 8, and their interactions of order two to eight, 12 , 13 , … , 12 ⋯ 8 [11]. Any three columns of the form ( a , b , a b ), where a b is the interaction column between columns a and b, can be used to represent the levels of the slice factor S without affecting orthogonality [1]. This replacement can be done according to the rule in Table 5.

Next, we discuss details of our design d with 2 6 + 2 − p runs. Consider a full factorial design with 2 6 + 2 − p runs, with the 4-level column represented by S = ( s 1 , s 2 , s 3 ) with s 3 = s 1 s 2 , and the 2-level columns represented by 1 , … , 6 − p. The remaining p columns, 6 − p + 1 , … , 6, can be generated as interactions of the first 6 − p + 2 columns. How to pick these p columns determines the generators and the defining relation of the design d. For a two-platform experiment, [8] defined the sliced wordlength pattern to accommodate the aliasing relation of the slice factor S. For a four-platform experiment, this definition does not work as the slice factor S has three aliasing relations of s 1 , s 2 , and s 3 , respectively. The aliasing relation of s j is obtained by multiplying the defining relation of d by s j . Therefore, a word W in the defining relation of d appears in the three aliasing relations for the slice factor S as s 1 W, s 2 W, and s 3 W. We extend [8]’s definition of the sliced wordlength pattern to cover the minimum length of s 1 W, s 2 W, and s 3 W. This extension minimizes the number of the shortest length of a sliced wordlength pattern. Defining the sliced wordlength pattern over the minimum length of s 1 W, s 2 W, and s 3 W ensures that the minimum sliced aberration protects against the worst-case scenario.

We use [11]’s definition of wordlength pattern for designs with two-level and four-level factors to define the sliced wordlength pattern. The design d with 2 6 + 2 − p runs has two types of words in its defining relation. The first, called type 0, involves only the design factors 1 , … , 6, and the second, called type 1, involves one of the s j ’s and some of the design factors 1 , … , 6. Because of the relation s 1 s 2 s 3 = I, any two s j ’s appearing in a word can be replaced by the third s j . Therefore, these two types cover all the possible combinations. Following [11], the vector (3.1) W ( d ) = ( [ A i 0 ( d ) , A i 1 ( d ) ] ) i ≥ 3 , is the wordlength pattern of d in which A i 0 ( d ) and A i 1 ( d ) are the numbers of type 0 and type 1 words of length i in the defining relation of d, respectively. The term [ A 20 ( d ) , A 21 ( d ) ] is not considered in (3.1) because any design d with a positive value of [ A 20 ( d ) , A 21 ( d ) ] is not useful as two of its main effects are aliased. We define the sliced wordlength pattern of a design d for a four-platform experiment as follows: For a design d with the wordlength pattern W ( d ) = ( [ A i 0 ( d ) , A i 1 ( d ) ] ) i ≥ 3 for the four-platform experiment under consideration, we define the sliced wordlength pattern to be the vector S W ( d ) = ( [ S A i 0 ( d ) , S A i 1 ( d ) ] ) i ≥ 2 , where

A type 0 word W in the defining relation of d appears as a type 1 word in the aliasing relations of the sliced factor S. It is counted as a type 1 word in the sliced wordlength pattern, resulting in S A i 1 ( d ) = A ( i − 1 ) 0 ( d ). A type 1 word W in the defining relation of d appears as a type 1 word in the aliasing relations of two s j ’s and as a type 0 word in the aliasing relation of the third s j . It is counted as a type 0 word in the sliced wordlength pattern with S A i 0 ( d ) = A ( i + 1 ) 1 ( d ) because the sliced wordlength pattern is defined over the minimum length of a word in the three aliasing relations.

The sliced resolution of d is defined to be the smallest i for which at least one of S A i 0 ( d ) and S A i 1 ( d ) is positive. Additional discrimination among designs with the same sliced resolution is covered by the following minimum sliced aberration. The two types of words of the design d are not treated the same. According to the sliced effect hierarchy, a type 1 word in the aliasing relations of the slice factor S is more serious because it involves one s j . This is consistent with [11]’s result that ranks a type 0 word in the defining relation of d more important than a type 1 because a type 0 word in the defining relation appears as a type 1 word in the aliasing relations of the slice factor S. Therefore, it is more important to require a smaller S A i 1 ( d ) than a smaller S A i 0 ( d ) for the same i. We define minimum sliced aberration designs for a four-platform experiment as follows:

For our experiment with six design factors, let s 1 , s 2 , 1, 2, and 3 be the five independent columns of the 32-run 2 5 design. Consider two designs: d ( 1 ) : S , 1 , 2 , 3 , 12 , 13 , 23 d ( 2 ) : S , 1 , 2 , 3 , 13 s 2 , 23 s 2 , 123 s 1 where S = ( s 1 , s 2 , s 3 = s 1 s 2 ) and the last three columns represent the last three design factors. For example, 4 = 12, 5 = 13 and 6 = 23 in d ( 1 ) and 4 = 13 s 2 , 5 = 23 s 2 and 6 = 123 s 1 in d ( 2 ) . Therefore, the defining relations of d ( 1 ) and d ( 2 ) are: d ( 1 ) : I = 124 = 135 = 236 = 2345 = 1346 = 1256 = 456 d ( 2 ) : I = 134 s 2 = 235 s 2 = 1236 s 1 = 1245 = 246 s 3 = 156 s 3 = 3456 s 1 . The defining relation of d ( 1 ) has seven words of type 0: four of length three and three of length four. The wordlength pattern of d ( 1 ) is W ( d ( 1 ) ) = ( [ 4 , 0 ] 3 , [ 3 , 0 ] 4 ). Multiplying the defining relation of d ( 1 ) by s j ’s provides the following three aliasing relations of the slice factor S: s 1 = 124 s 1 = 135 s 1 = 236 s 1 = 2345 s 1 = 1346 s 1 = 1256 s 1 = 456 s 1 s 2 = 124 s 2 = 135 s 2 = 236 s 2 = 2345 s 2 = 1346 s 2 = 1256 s 2 = 456 s 2 s 3 = 124 s 3 = 135 s 3 = 236 s 3 = 2345 s 3 = 1346 s 3 = 1256 s 3 = 456 s 3 . Each type 0 word in the defining relation of d ( 1 ) appears as type 1 word in all three aliasing relations of s j ’s. The sliced wordlength pattern of d ( 1 ) is S W ( d ( 1 ) ) = ( [ 0 , 0 ] 2 , [ 0 , 0 ] 3 , [ 0 , 4 ] 4 , [ 0 , 3 ] 5 ). The defining relation of d ( 2 ) has one word of type 0 of length four and six words of type 1: four of length four and two of length five. The wordlength pattern of d ( 2 ) is W ( d ( 2 ) ) = ( [ 0 , 0 ] 3 , [ 1 , 4 ] 4 , [ 0 , 2 ] 5 ). Multiplying the defining relation of d ( 2 ) by s j ’s provides the following three aliasing relations of the slice factor S: s 1 = 134 s 3 = 235 s 3 = 1236 _ = 1245 s 1 = 246 s 2 = 156 s 2 = 3456 _ s 2 = 134 _ = 235 _ = 1236 s 3 = 1245 s 2 = 246 s 1 = 156 s 1 = 3456 s 3 s 3 = 134 s 1 = 235 s 1 = 1236 s 2 = 1245 s 3 = 246 _ = 156 _ = 3456 s 2 . The type 1 word 134 s 2 in the defining relation of d ( 2 ) appears as type 0 word of length three in the aliasing relation of s 2 because s 2 s 2 = I and as type 1 word of length four in the aliasing relations of s 1 and s 3 because s 1 s 2 = s 3 and s 3 s 2 = s 1 . It has length three of type 0 in the sliced wordlength pattern. Similar explanations can be used for the other six words in the defining relation of d ( 2 ) . The sliced wordlength pattern of d ( 2 ) is S W ( d ( 2 ) ) = ( [ 0 , 0 ] 2 , [ 4 , 0 ] 3 , [ 2 , 0 ] 4 , [ 0 , 1 ] 5 ). Between the two designs d ( 1 ) and d ( 2 ) , 3 is the smallest integer such that [ 0 , 0 ] 3 ( d ( 1 ) ) ≠ [ 4 , 0 ] 3 ( d ( 2 ) ). The design d ( 1 ) has less sliced aberration than d ( 2 ) because S A 31 ( d ( 1 ) ) = S A 31 ( d ( 2 ) ) = 0 and S A 30 ( d ( 1 ) ) = 0 < 4 = S A 30 ( d ( 2 ) ). We will show later that d ( 1 ) is a minimum sliced aberration design with six design factors and 32 runs. Here d ( 2 ) is a minimum aberration design with 32 runs from [11], which is inferior to a minimum sliced aberration design for a four-platform experiment.

Equipped with a suitable design criterion for our experiment, we are now ready to construct the corresponding minimum sliced aberration designs given in Section 2. Theorem 1 below guides the construction of the minimum sliced aberration designs using readily available minimum aberration designs of fewer numbers of factors.

As a result of Theorem 1, constructing a minimum sliced aberration design entails a search among possible designs for which all the words are type 0 in the defining relation. Therefore, minimizing the number of the shortest length in the sliced wordlength pattern of d with 2 6 + 2 − p runs is equivalent to minimizing the number of the shortest length in the wordlength pattern of a 2 6 − p fractional design consisting of design factors only. We use Theorem 1 to generate the minimum sliced aberration design given in Table 6.

The minimum sliced aberration designs in Theorem 1 have a cross array structure similar to product parameter design [10].

The aforementioned theoretical results and the construction method work for the general k number of factors by changing six to k. Following the general case of Theorem 1, constructing a sliced minimum aberration design entails search among possible designs for which all the words are type 0 in the defining relation. Therefore, minimizing the number of the shortest length in the sliced wordlength pattern of d with 2 k + 2 − p runs is equivalent to minimizing the number of the shortest length in the wordlength pattern of a 2 k − p fractional design consisting of design factors only. For a four-platform experiment, we use Theorem 1 to provide sliced minimum aberration designs with 16, 32, and 64 runs in Tables 7-9, respectively.

Using the design criterion in Section 3, we create 2 3 versions to perform the multivariate testing. Each platform has eight versions and we generate them based on the criterion in Section 3 to perform this multivariate testing. This design is a 2 6 + 2 − 3 minimum sliced aberration design for four platforms. By Theorem 1, the sub design for each platform is a 2 6 − 3 minimum aberration design with the three generators 124, 135, and 236 [10, p.252]. The sliced word length pattern is ( [ 0 , 4 ] 4 , [ 0 , 3 ] 5 ). More details of this design were discussed in Section 2.

The response variable in the study is the email open rate. As the data are aggregated across users exposed to each version, how the response variable varies within a version is unknown to us. We use the Lenth’s method [6] to identify significant factors, which is specifically designed for testing effects in experiments for which variance estimates are not available.

Table 10 includes some descriptive statistics of the study. The total number of recipients is 139033, which are divided into roughly equal eight sets receiving the eight versions of the email. Table 11 is a two-way table providing the number of opened emails in each combination of operating systems and email versions.

Since there are eight versions for each operating system (platform), seven effects of design factors can be estimated per platform. Table 12 includes the aliased effects within each platform. For convenience, we label each set of aliased effects.

In the sliced factorial design framework, slices are used for analyzing data of the four operating systems together. Tables 13 to 16 include the effects of design factors that are estimated using the design in Table 2 on each platform. The Lenth’s method is used to test the significance of effects and to report the p-values. The same method is done for each operating system to estimate the effect of design factors within the platform.

Comparing Tables 13-16 indicates that effect B is significant on P 1 (Android), two effects B and F are significant on P 2 (iOS), and effect D is significant on P 4 (macOS) although no effect is significant on P 3 (Windows). Table 12 reveals that effect B is the sum of the following aliased effects 2 , 14 , 36 , 345 , 156 , 2456 , 1235 , 12346. As the slices follow the effect hierarchy principle, B can be viewed to represent effect 2 by assuming that all higher-order aliased effects are negligible. The main takeaway for the Android system from Table 13 is that using the direct subject line will likely decrease the open rate and other factors are not expected to decrease or increase the metric. Similar arguments can be made for the other three platforms P 2 , P 3 , and P 4 . For P 2 from Table 14, using the direct subject line and displaying content in paragraph is expected to decrease the open rate. The remaining factors will not likely impact the metric in any way. The main takeaway for the Windows system from Table 15 is that no factor will likely affect the metric. The main takeaway for the macOS system from Table 16 is that including header image is expected to decrease the open rate and other factors are not expected to have any effect on the response. In summary, a comparison of Tables 13 to 16 indicates that different sets of design factors work differently for these multiple operating systems.

From Table 11, the open rate for Windows is extremely larger than that of other platforms. As no factor will affect the open rate for the Windows system, the operating system might impact the metric. It is important to figure out whether different operating systems have significant interaction with the platform-specific significant effects. To compare the results of the four platforms, the complete design d is used to estimate the slice factor and its interaction with the platform-specific significant effects. The slice factor S is represented by S = ( s 1 , s 2 , s 3 ) with s 3 = s 1 s 2 . Table 17 describes the relation between P i ’s and ( s 1 , s 2 , s 3 ).

We use Lenth’s method to test the significance of the effects. Because the slice factor S is four-level, the effect S contains three effects s 1 , s 2 and s 3 . Take the interaction between S and 2 for example. It includes three effects 2 s 1 , 2 s 2 , and 2 s 3 . Table 18 includes the 12 effects for S and the interactions between the slice factor S and three platform-specific significant effects 2, 4, and 6. We now focus on the first three effects s 1 , s 2 , and s 3 in Table 18. The magnitude of the effect of Android is 2.41e-2, the counterpart of the effect of iOS is 0.79e-2, the counterpart of Windows is 5.01e-2, and the counterpart of macOS is 1.81e-2. The magnitude of the effect of Windows is about two times larger than the effect of Android, six times larger than the effect of iOS, and three times larger than the effect of macOS, which explains why the open rate for Windows is extremely larger than that of other platforms. Further, the magnitude of the effects s 1 , s 2 , and s 3 are around ten to hundred times larger than the effects of design factors. This finding is consistent with the sliced effect hierarchy principle. The other effects except for s 1 , s 2 and s 3 in Table 18 uncover the way the effects 2, 4, and 6 interact with the slice factor S, respectively, meaning how these effects differentially affect the open rate from P 1 to P 4 . Only 6 s 3 is significant, implying that the differential effect of 6 on the open rate from P 2 and P 3 to P 1 and P 4 is significant. As a result, one version that adopts displaying content in paragraph is expected to decrease the open rate in P 2 and P 3 . However, the same version will likely increase the metric in P 1 and P 4 .

In conclusion, the application of the sliced design in Section 3 to the email campaign shows that different sets of design factors would increase the open rate for each of the four operating systems. From Tables 13 to 16, each design factor significant on the corresponding platform has a negative effect, which means factors 2, 4, and 6 should be at the − level. See Table 1 for the information of the design factors. Because we do not know in advance what platform a particular user will use to open the email, it is desirable to choose a version that has factors 2, 4, and 6 at the − level. This hypothesis should be tested by looking at the interactions between S and the three design factors 2, 4, and 6 from the complete design d. Table 18 indicates the effect of 6 s 3 is significant, implying that 6 at the − level is expected to increase the open rate in P 2 and P 3 but would decrease the metric in P 1 and P 4 . In order to test this hypothesis, we fit a regression model using the open rate as the response and s 1 , s 2 , s 3 , 2, 4, 6, and 6 s 3 as the covariates. The average open rate is estimated by (4.1) Average Open Rate = 0.0163 + 0.0080 s 1 − 0.0065 s 2 − 0.0105 s 3 − 0.0008 B − 0.0005 D − 0.0010 F + 0.0009 F s 3 .

We use (4.1) to compare the average open rate between the two versions: design factors 2, 4, 6 are at − level and design factors 2, 4 are at − level but 6 is at + level for each of the four platforms. The result is given in Table 19.

Table 19 indicates that the average open rate of version A is larger than that of version B for each of the four operating systems. To conclude, we recommend the following changes for the network company to increase the open rate: (i) use the indirect subject line, (ii) drop the header image, and (iii) display content in bullet points. These statistics-guided recommendations can help the network company optimize email campaigns and increase the ROI of its marketing efforts.