Bayes Factors for Comparison of Two-Way ANOVA Models

Inthetraditionaltwo-wayanalysisofvariance(ANOVA)model,itispossibletoidentifythesignificanceofboththemaineffects andtheirinteractionbasedonthe P values. However, it is not possible to determine how much data supports the model when these effects are incorporated into the model. To overcome this practical difficulty, we applied Bayes factors for hierarchical models to check the intensity of the effects (both main and interaction). The objective is to identify the impact of the main and interaction effects based on a comparison of Bayes factors of the hierarchical ANOVA models. The application of Bayes factors enables to observe which model strengthens more while including or eliminating the effects in the model. Consequently, this paper proposes three priors such as Zellner’s g , Jefferys-Zellner-Siow, and Hyper-g priors, to compute the Bayes factor. Finally, we extended this procedure to the simulation data for the generalization of the Bayesian results.


INTRODUCTION
The Bayesian method is widely applied in a range of disciplines such as engineering, medicine, biology, economics, etc. Jeffrey [1] developed the Bayesian approach to solve hypothesis testing problems.He also developed a methodology for quantifying the evidence in favor of a scientific theory.Kass and Raftery [2] discussed the Bayesian viewpoints and several techniques for computing Bayes factor.Bayesian approach to the analysis of variance (ANOVA) model was discussed by many authors (Liang et al. [3]; Maruyama and George [4]; Wetzels et al. [5], etc.).In this study, we aim to identify the influence of the effects when we incorporate eliminating from the ANOVA two-way models by the Bayesian approach.The main advantage of this approach is that the evidence is quantified, instead of forcing an all or none decision as in the frequentist approach.The Bayes factor results are quantified in Table 1.Several priors are discussed to find the Bayes factor by many authors, Liang et al. [3], Wang and Sun [6], Kass and Raftery [2], Clyde et al. [7].

Zellner's g Prior
Zellner's g priors are most commonly used prior for Bayesian hypothesis testing.The limit of inverse-gamma distribution with shape and scale used as prior density.Many authors, Kass and Raftery [2], Zellner [8], and others discussed extensively this prior.Its modified version was introduced by Liang et al. [3].That is, the g prior is assigned to all the regression coefficients except the intercept.We consider two priors by setting the value of g, (i) Unit Information Prior (UIP) if g = n and (ii) Risk Inflation Criterion (RIC) if g = k 2 , where n = number of observations and k = the number of predictors in the regression model.The Bayes factor for the full model to the null model is (1)

Jeffrey-Zellner-Siow Prior
Jeffrey-Zellenr-Siow (JZS) prior is a mixture of priors, such that, Jeffreys' prior on the intercept and an inverse gamma with ½ and n/2 prior on g.Further, estimate g from the data and the Bayes factor is not analytically available, Liang et al. [3], Ley et al. [9].The Bayes Factor for

Hyper-g Prior
Hyper-g prior is a family of prior distributions on g and this hyper-g approach p(g) The range of values for a lies in between 2 and 4. It produces different behavior of hyper-g prior, generally, its proposed values are a = 3 and/or a = 4, apart from that the issue of arbitrary constants of proportionality leads to indeterminate Bayes factors.The Bayes factor for the full model to the null model is To avoid the ambiguous conclusions of the Bayesian approach, we propose three different priors such as Zellner's g, Jeffreys-Zellner-Siow, and Hyper-g priors, Wetzels et al. [5], Perrakis and Ntzoufras [10], Vijayaragunathan and Srinivasan [11].The level of interpretations for the Bayes factor for the null and alternative model are in Table 1.The results are quantified evidence of data support the respective model, which was discussed extensively by Jeffreys [1] and Kass and Raftery [10].

ANOVA TWO-WAY MODELS AND BAYESIAN APPROACH
Factor A has a levels, a factor B has b levels, and these factors are arranged in a factorial design each replicate of the experiment contains all ab treatment combinations.An ANOVA two-factor experimental design is where  is the overall mean effect,  i is the effect of ith level of factor A,  j is the effect of jth level of factor B, () ij is the effect of the interaction between A and B, and  ijk is a random error component.
The two-way ANOVA model is converted into a regression framework model for our convenience in applying the Bayesian technique to different priors Zellner et al. [13].To test the outcome of main effects A, and B and the interaction effects of A and B, by comparing four possible ANOVA designs.An example of a two-way ANOVA model is discussed in the next section.The main effect A is plate material has three levels P 1 , P 2 , and P 3 ; the second main effect B is the temperature also has three levels T 1 , T 2 , and T 3 ; and its interaction effect has nine possible combinations.The regression framework for the two-way ANOVA experiment is as follows: where Xis a design matrix formed according to the factors by using dummy codes.The design matrix is not necessarily a full column rank.
We adopt the sum-to zero constraints, so the regression coefficient of the last level of each factor is equal to the sum of the other regression coefficients.Therefore, the two-factor ANOVA experiment full model contains grand mean , two levels of main effect A, and two levels of the main effect B and it's interaction effects (the third level of A and B omitted because of sum-to-zero constraint) is The two-way ANOVA model containing only the main effects, without interaction effects, is Pdf_Folio:541 The two-way ANOVA model containing only the main effect A is The two-way ANOVA model containing only the main effect B is The two-way ANOVA model without main and interaction effects (the null model) is Let BF M i ∶M 0 or BF i0 , i = 1, 2, 3, and 4 , be the Bayes factor for the ith model to the null model Rouder et al. [5].The ANOVA Model 1 (full model) is a larger model to other models; it means that the reduced models are nested to the full model.We compare reduced models with the larger model to test the cause of the respective factors.To test an interaction effect, we want to find BF 12 that is Bayes factor for a larger model (model 1) to the reduced model (model 2).We have to compute the Bayes factor BF 10 and BF 20 to find BF 12, then the ratio of these Bayes factors is because Bayes factors are inverse and transitivity.Similarly, the ratio of Bayes factors BF 23 and BF 24 are found by using BF 20 & BF 30 and BF 20 & BF 40 respectively to test the effect of main effects.It does not give any assertion about the impact of main and interaction effects in the model based on a single prior.Therefore, we proposed three different priors to find the Bayes factor.In the next session, we will discuss on Bayes factors of these priors.In general, it couldn't take any legitimate decision based on the simple application.Further, we decided to find the number of Bayes factors from the simulation datasets, which is discussed in the fourth section.

APPLICATIONS
Consider an engineer is designing a battery for use in a device that will be subject to some extreme variations in temperature, and he has three possible choices of plate material [14].The engineer has no control over the temperature extremes that the device will encounter, and he knows from experience that temperature will probably affect the effectiveness of battery life.He decides that all three plate materials at three temperature levels (15∘F, 70∘F, and 125∘F) because these temperature levels are consistent with the product end-use environment.Four batteries were tested at each combination of plate material and temperature and all 36 tests are run in random order.The F ratio and corresponding P values of four ANOVA models have presented in Table 2.The main effect of A (plate material) is significant in models 1 and 2, but in the single treatment model 3, there is no significant difference among the average battery life of different plate material.Thus, the effect of plate material would be significant only if it is with the other effect.The main effect of B (temperature) is strongly significant in all models when it exists.Also, the interaction effect between the plate material and temperature is moderately significant.
In the first step of the Bayesian approach, we have computed the Bayes Factor for all ANOVA models with the null model, to know how the data support the respective models.From Table 3, the evidence of data supports "decisively" for the two-way ANOVA models such as model 1 (full model), model 2 (except interaction effects model), and model 4 (only main effect B). But, the Bayes factor for model 3 (only the main effect A) does not provide impressive results, such that two priors are "poorly" support the null model and the other three priors are almost "poorly" support model 3.
The two-way ANOVA with interaction effect model (full model) is a larger model, other models are nested models to it, and so on.Now, to test the effect of interaction by comparing the full model and without the interaction model.In the same way, we used the without interaction model and respective main effect models to test the main effects.In general, the Bayes factors for the larger model to the nested model have been computed to know the strength of the effect which was eliminated in nested models.
The ratio of Bayes factors of the full model and without interaction models indicate that there is an intensity of the interaction effect in the model.The Bayes factor values are close to one for BF (12) and BF(24), which gives the impression that the data support "strongly" for both interaction and main effect B (temperature) of battery life.All the priors support the respective models invariably for these effects.But, for the main effect, A (plate material) model provides notable results, the UIP and JZS priors are "strongly" support model 3 and the other three priors are "strongly" support model 2. Because, the main effect A is significant when it is together with the main effect B and nonsignificant when it is alone in the ANOVA model, are perhaps the reason for the variation in results.We cannot trust the applications with a single Bayes factor value, therefore, in the next section, we will demonstrate several Bayes factors in the simulation study to generalize the interpretations.

SIMULATION STUDY
In an examination of the effects of the two-way ANOVA model, we constructed the different datasets by varying different error variance in the simulation.To find the Bayes Factors for all priors, we have to simulate datasets to obtain reliable conclusions.Firstly, we simulated 1000 different data sets with variance 1 to compute the 1000 Bayes factor values for each prior.Similarly, we simulated the other two datasets with the error variances 10 and 25 to compute Bayes factors.These results are shown in Figures 1-5 and also display the summary statistics for the Bayes factor of respective simulation data to the ANOVA model are shown in Table 4-9.The overall conclusion of all the priors do not change in BF (12) and BF(24), but varies in BF(23).Thus, our simulation study provides broad knowledge about the different priors of various datasets.Consequently, it may be useful to make decisions for inclusion or elimination of the interaction effect in the two-way ANOVA model as well as the influence of the main effects in the model.

CONCLUSIONS
An ANOVA is one of the most popularly used statistical tools in research.We considered a two-way ANOVA model consisting of two factors such as plate material (A) and temperature (B) related to battery life.In the classical approach of the ANOVA two-way model, we examined and discussed the significance of main and interaction effects as usual.We also examined the contribution level of effect(s) in the two-way ANOVA model by the Bayes factor of choice of different priors.In Bayesian Approach, it is found that the Bayes factors for all four ANOVA Pdf_Folio:544  models with the null model to know the strength of interaction and main effects in the respective models.From the simulation study, the Bayes factor results provide ample evidence for support of the data "strongly" both the models (with the interaction effect and main effect B (temperature)).For these effects, all the priors support the respective models invariably.Nevertheless, the main effect A (plate material) model provides "strongly" support to model 2 as well as model 3. We may deduct that the main effect A is significant only when it is together with the main effect B and nonsignificant when it is alone in the ANOVA model.All priors generate almost similar average Bayes factor values with different standard deviations for the models consisting of interaction and the main effect of temperature.It is observed that the same priors do not generate the same results in the model consisting of the main effect of plate material.Thus, we conclude that there is no single correct prior distribution and if we use different priors it may provide more instructive conclusions.The simulation technique may be highly rewarding to the researchers if properly applied in their respective areas.

Figure 1
Figure 1 Bayes factors of simulation data sets for different variances to the Zellner's prior (unit information prior).

Figure 2
Figure 2 Bayes factors of simulation data sets for different variances to the Zellner's g prior (risk inflation criterion).

Figure 3
Figure 3 Bayes factors of simulation datasets for different variances to the Jeffreys-Zellner-Siow prior.

Figure 4
Figure 4 Bayes factors of simulation datasets for different variances to the hyper-g (a = 3) prior.

Figure 5
Figure 5 Bayes factors of simulation datasets for different variances to the hyper-g (a = 4) prior.

Table 1
Interpretations of Bayes factor.

Table 2 F
ratio for four ANOVA models (* represents P values).

Table 3
Bayes factor for respective two-way ANOVA model with the null model.

Table 4
Ratio of Bayes factors for the relevant ANOVA models.

Table 5
Summary of Bayes factors for Zellner's g prior (unit information prior).

Table 6
Summary of Bayes factors for Zellner's g prior (risk inflation criterion).

Table 7
Summary of Bayes factors for Jeffreys-Zellner-Siow prior.

Table 8
Summary of Bayes factors for hyper-g (a = 3) prior.

Table 9
Summary of Bayes factors for hyper-g (a = 4) prior.