- Equal groups sizes: Tukey
- Unequal groups sizes: Tukey, Scheffe
- with control: Dunnett
- general: Bonferroni, Scheffe
21.1.2 Single Factor Random Effects Model
Also known as ANOVA Type II models.
Treatments are chosen at from from larger population. We extend inference to all treatments in the population and not restrict our inference to those treatments that happened to be selected for the study.
21.1.2.1 Random Cell Means
- \(\mu_i \sim N(\mu, \sigma^2_<\mu>)\) and independent
- \(\epsilon_ \sim N(0,\sigma^2)\) and independent
\(\mu_i\) and \(\epsilon_\) are mutually independent for \(i =1. a; j = 1. n\)
With all treatment sample sizes are equal
\[ \begin E(Y_) &= E(\mu_i) = \mu \\ var(Y_) &= var(\mu_i) + var(\epsilon_i) = \sigma^2_ <\mu>+ \sigma^2 \end \]
Since \(Y_\) are not independent
\[ \begin cov(Y_,Y_) &= E(Y_Y_) - E(Y_)E(Y_) \\ &= E(\mu_i^2 + \mu_i \epsilon_ + \mu_i \epsilon_ + \epsilon_\epsilon_) - \mu^2 \\ &= \sigma^2_ <\mu>+ \mu^2 - \mu^2 & \text j \neq j' \\ &= \sigma^2_ <\mu>& \text j \neq j' \end \]
\[ \begin cov(Y_,Y_) &= E(\mu_i \mu_ + \mu_i \epsilon_+ \mu_\epsilon_+ \epsilon_\epsilon_) - \mu^2 \\ &= \mu^2 - \mu^2 & \text i \neq i' \\ &= 0 \\ \end \]
- all observations have the same variance
- any two observations from the same treatment have covariance \(\sigma^2_<\mu>\)
- The correlation between any two responses from the same treatment:
\[ \begin \rho(Y_,Y_) &= \frac<\sigma^2_<\mu>><\sigma^2_<\mu>+ \sigma^2> && \text \end \]
Inference
Intraclass Correlation Coefficient
which measures the proportion of total variability of \(Y_\) accounted for by the variance of \(\mu_i\)
\[ \begin &H_0: \sigma_<\mu>^2 = 0 \\ &H_a: \sigma_<\mu>^2 \neq 0 \end \]
\(H_0\) implies \(\mu_i = \mu\) for all i, which can be tested by the F-test in ANOVA.
The understandings of the Single Factor Fixed Effects Model and the Single Factor Random Effects Model are different, the ANOVA is same for the one factor model. The difference is in the expected mean squares
| Random Effects Model | Fixed Effects Model |
|---|---|
| \(E(MSE) = \sigma^2\) | \(E(MSE) = \sigma^2\) |
| \(E(M STR) = \sigma^2 - n \sigma^2_\mu\) | \(E(MSTR) = \sigma^2 + \frac< \sum_i n_i (\mu_i - \mu)^2>\) |
If \(\sigma^2_\mu\) , then MSE and MSTR have the same expectation ( \(\sigma^2\) ). Otherwise, \(E(MSTR) >E(MSE)\) . Large values of the statistic
suggest we reject \(H_0\) .
Since \(F \sim F_\) when \(H_0\) holds. If \(F > f_\) we reject \(H_0\) .
If sample sizes are not equal, \(F\) -test can still be used, but the df are \(a-1\) and \(N-a\) .
21.1.2.1.1 Estimation of \(\mu\)
An unbiased estimator of \(E(Y_)=\mu\) is the grand mean: \(\hat <\mu>= \hat_\)
The variance of this estimator is
An unbiased estimator of this variance is \(s^2(\bar)=\frac\) . Thus \(\frac<\bar_-\mu>
A \(1-\alpha\) confidence interval is \(\bar_ \pm t_<(1-\alpha/2;a-1)>s(\bar_)\)
21.1.2.1.2 Estimation of \(\sigma^2_\mu/(\sigma^2_<\mu>+\sigma^2)\)
In the random and fixed effects model, MSTR and MSE are independent. When the sample sizes are equal ( \(n_i = n\) for all i),
If the lower limit for \(\frac<\sigma^2_\mu>\) is negative, it is customary to set \(L = 0\) .
21.1.2.1.3 Estimation of \(\sigma^2\)
\(a(n-1)MSE/\sigma^2 \sim \chi^2_\) , the \((1-\alpha)\) confidence interval for \(\sigma^2\) :
can also be used in case sample sizes are not equal - then df is N-a.
21.1.2.1.4 Estimation of \(\sigma^2_\mu\)
\(E(MSE) = \sigma^2\) \(E(MSTR) = \sigma^2 + n\sigma^2_\mu\) . Hence,
An unbiased estimator of \(\sigma^2_\mu\) is given by
If sample sizes are not equal,
no exact confidence intervals for \(\sigma^2_\mu\) , but we can approximate intervals.
Satterthewaite Procedure can be used to construct approximate confidence intervals for linear combination of expected mean squares
A linear combination:
\[ \sigma^2_\mu = \frac E(MSTR) + (-\frac) E(MSE) \]
\[ S = d_1 E(MS_1) + ..+ d_h E(MS_h) \]
where \(d_i\) are coefficients.
An unbiased estimator of S is
\[ \hat = d_1 MS_1 + . + d_h MS_h \]
Let \(df_i\) be the degrees of freedom associated with the mean square \(MS_i\) . The Satterthwaite approximation:
An approximate \(1-\alpha\) confidence interval for S:
For the single factor random effects model
21.1.2.2 Random Treatment Effects Model
\[ \tau_i = \mu_i - E(\mu_i) = \mu_i - \mu \]
we have \(\mu_i = \mu + \tau_i\) and
\[ Y_ = \mu + \tau_i + \epsilon_ \]
- \(\mu\) = constant, common to all observations
- \(\tau_i \sim N(0,\sigma^2_\tau)\) independent (random variables)
- \(\epsilon_ \sim N(0,\sigma^2)\) independent.
- \(\tau_, \epsilon_\) are independent (i=1,…,a; j =1. n)
- our model is concerned with only balanced single factor ANOVA.
Diagnostics Measures
- Non-constant error variance (plots, Levene test, Hartley test).
- Non-independence of errors (plots, Durban-Watson test).
- Outliers (plots, regression methods).
- Non-normality of error terms (plots, Shapiro-Wilk, Anderson-Darling).
- Omitted Variable Bias (plots)
Remedial
- Weighted Least Squares
- Transformations
- Non-parametric Procedures.
Note
- Fixed effect ANOVA is relatively robust to
- non-normality
- unequal variances when sample sizes are approximately equal; at least the F-test and multiple comparisons. However, single comparisons of treatment means are sensitive to unequal variances.
21.1.3 Two Factor Fixed Effect ANOVA
The multi-factor experiment is
- more efficient
- provides more info
- gives more validity to the findings.
21.1.3.1 Balanced
- All treatment sample sizes are equal
- All treatment means are of equal importance
- Factor \(A\) has a levels and Factor \(B\) has b levels. All \(a \times b\) factor levels are considered.
- The number of treatments for each level is n. \(N = abn\) observations in the study.
21.1.3.1.1 Cell Means Model
- \(\mu_\) are fixed parameters (cell means)
- \(i = 1. a\) = the levels of Factor A
- \(j = 1. b\) = the levels of Factor B.
- \(\epsilon_ \sim \text N(0,\sigma^2)\) for \(i = 1. a\) , \(j = 1. b\) and \(k = 1. n\)
And the model is
\[ \mathbf = \mathbf \beta + \epsilon \]
\[ \begin E(\mathbf) &= \mathbf\beta \\ var(\mathbf) &= \sigma^2 \mathbf \end \]
Interaction
\[ (\alpha \beta)_ = \mu_ - (\mu_+ \alpha_i + \beta_j) \]
- \(\mu_ <..>= \sum_i \sum_j \mu_/ab\) is the grand mean
- \(\alpha_i = \mu_-\mu_<..>\) is the main effect for factor \(A\) at the \(i\) -th level
- \(\beta_j = \mu_ - \mu_<..>\) is the main effect for factor \(B\) at the \(j\) -th level
- \((\alpha \beta)_\) is the interaction effect when factor \(A\) is at the \(i\) -th level and factor \(B\) is at the \(j\) -th level.
- \((\alpha \beta)_ = \mu_ - \mu_-\mu_+ \mu_<..>\)
- Examine whether all \(\mu_\) can be expressed as the sums \(\mu_ <..>+ \alpha_i + \beta_j\)
- Examine whether the difference between the mean responses for any two levels of factor \(B\) is the same for all levels of factor \(A\) .
- Examine whether the difference between the mean response for any two levels of factor \(A\) is the same for all levels of factor \(B\)
- Examine whether the treatment mean curves for the different factor levels in a treatment plot are parallel.
\[ \begin \sum_i(\alpha \beta)_ &= \sum_i (\mu_ - \mu_ - \alpha_i - \beta_j) \\ &= \sum_i \mu_ - a \mu_ - \sum_i \alpha_i - a \beta_j \\ &= a \mu_ - a \mu_- \sum_i (\mu_ - \mu_) - a(\mu_-\mu_) \\ &= a \mu_ - a \mu_ - a \mu_+ a \mu_ - a (\mu_ - \mu_) \\ &= 0 \end \]
Similarly, \(\sum_j (\alpha \beta) = 0, i = 1. a\) and \(\sum_i \sum_j (\alpha \beta)_ =0\) , \(\sum_i \alpha_i = 0\) , \(\sum_j \beta_j = 0\)
21.1.3.1.2 Factor Effects Model
\[ \begin \mu_ &= \mu_ + \alpha_i + \beta_j + (\alpha \beta)_ \\ Y_ &= \mu_ + \alpha_i + \beta_j + (\alpha \beta)_ + \epsilon_ \end \]
- \(\mu_<..>\) is a constant
- \(\alpha_i\) are constants subject to the restriction \(\sum_i \alpha_i=0\)
- \(\beta_j\) are constants subject to the restriction \(\sum_j \beta_j = 0\)
- \((\alpha \beta)_\) are constants subject to the restriction \(\sum_i(\alpha \beta)_ = 0\) for \(j=1. b\) and \(\sum_j(\alpha \beta)_ = 0\) for \(i = 1. a\)
- \(\epsilon_ \sim \text N(0,\sigma^2)\) for \(k = 1. n\)
\[ \begin E(Y_) &= \mu_ + \alpha_i + \beta_j + (\alpha \beta)_\\ var(Y_) &= \sigma^2 \\ Y_ &\sim N (\mu_ + \alpha_i + \beta_j + (\alpha \beta)_, \sigma^2) \end \]
We have \(1+a+b+ab\) parameters. But there are \(ab\) parameters in the Cell Means Model. In the Factor Effects Model, the restrictions limit the number of parameters that can be estimated:
\[ \begin 1 &\text < for >\mu_ \\ (a-1) &\text < for >\alpha_i \\ (b-1) &\text < for >\beta_j \\ (a-1)(b-1) &\text < for >(\alpha \beta)_ \end \]
Hence, there are
\[ 1 + a - 1 + b - 1 + ab - a- b + 1 = ab \]
parameters in the model.
We can have several restrictions when considering the model in the form \(\mathbf = \mathbf \beta + \epsilon\)
We can fit the model by least squares or maximum likelihood
Cell Means Model
minimize\[ Q = \sum_i \sum_j \sum_k (Y_-\mu_)^2 \]
Factor Effects Model
\[ Q = \sum_i \sum_j \sum_k (Y_ - \mu_-\alpha_i = \beta_j - (\alpha \beta)_)^2 \]
subject to the restrictions
\[ \begin \sum_i \alpha_i &= 0 \\ \sum_j \beta_j &= 0 \\ \sum_i (\alpha \beta)_ &= 0 \\ \sum_j (\alpha \beta)_ &= 0 \end \]
The fitted values
21.1.3.1.2.1 Partitioning the Total Sum of Squares
\(Y_ - \bar_<. >\) : Total deviation
\(\bar_ - \bar_<. >\) : Deviation of treatment mean from overall mean
\(Y_ - \bar_\) : Deviation of observation around treatment mean (residual).\[ \begin \sum_i \sum_j \sum_k (Y_ - \bar_<. >)^2 &= n \sum_i \sum_j (\bar_- \bar_<. >)^2+ \sum_i \sum_j sum_k (Y_ - \bar)^2 \\ SSTO &= SSTR + SSE \end \]
(cross product terms are 0)
squaring and summing:
\[ \begin n\sum_i \sum_j (\bar_-\bar_<. >)^2 &= nb\sum_i (\bar_-\bar_<. >)^2 + na \sum_j (\bar_-\bar_<. >)^2 \\ &+ n \sum_i \sum_j (\bar_-\bar_- \bar_+ \bar_<. >)^2 \\ SSTR &= SSA + SSB + SSAB \end \]
The interaction term from
\[ \begin SSAB &= SSTO - SSE - SSA - SSB \\ SSAB &= SSTR - SSA - SSB \end \]
- \(SSA\) is the factor \(A\) sum of squares (measures the variability of the estimated factor \(A\) level means \(\bar_\) )- the more variable, the larger \(SSA\)
- \(SSB\) is the factor \(B\) sum of squares
- \(SSAB\) is the interaction sum of squares, measuring the variability of the estimated interactions.
21.1.3.1.2.2 Partitioning the df
\(N = abn\) cases and \(ab\) treatments.
For one-way ANOVA and regression, the partition has df:
\[ SS: SSTO = SSTR + SSE \]
we must further partition the \(ab-1\) df with SSTR
\[ SSTR = SSA + SSB + SSAB \]
- \(df_ = a-1\) : a treatment deviations but 1 df is lost due to the restriction \(\sum (\bar_- \bar_)=0\)
- \(df_ = b-1\) : b treatment deviations but 1 df is lost due to the restriction \(\sum (\bar_- \bar_)=0\)
- \(df_ = (a-1)(b-1)= (ab-1)-(a-1)-(b-1)\) : ab interactions, there are (a+b-1) restrictions, so df = ab-a-(b-1)= (a-1)(b-1)
21.1.3.1.2.3 Mean Squares
The expected mean squares are
If there are no factor A main effects (all \(\mu_ = 0\) or \(\alpha_i = 0\) ) the MSA and MSE have the same expectation; otherwise MSA > MSE. Same for factor B, and interaction effects. which case we can examine F-statistics.
Interaction
\[ \begin &H_0: \text(\alpha \beta)_ = 0 \\ &H_a: \text (\alpha \beta) = 0 \end \]
Factor A main effects:
21.1.3.1.2.4 Two-way ANOVA
Source of Variation SS df MS F Factor A \(SSA\) \(a-1\) \(MSA = SSA/(a-1)\) \(MSA/MSE\) Factor B \(SSB\) \(b-1\) \(MSB = SSB/(b-1)\) \(MSB/MSE\) AB interactions \(SSAB\) \((a-1)(b-1)\) \(MSAB = SSAB /MSE\) Error \(SSE\) \(ab(n-1)\) \(MSE = SSE/ab(n-1)\) Total (corrected) \(SSTO\) \(abn - 1\) Doing 2-way ANOVA means you always check interaction first, because if there are significant interactions, checking the significance of the main effects becomes moot.
The main effects concern the mean responses for levels of one factor averaged over the levels of the other factor. When interaction is present, we can’t conclude that a given factor has no effect, even if these averages are the same. It means that the effect of the factor depends on the level of the other factor.
On the other hand, if you can establish that there is no interaction, then you can consider inference on the factor main effects, which are then said to be additive.
And we can also compare factor means like the Single Factor Fixed Effects Model using Tukey, Scheffe, Bonferroni.We can also consider contrasts in the 2-way model
\[ L = \sum c_i \mu_i \]
where \(\sum c_i =0\)
which is estimated byand variance estimate
Orthogonal Contrasts
\[ \begin L_1 &= \sum c_i \mu_i, \sum c_i = 0 \\ L_2 &= \sum d_i \mu_i , \sum d_i = 0 \end \]
these contrasts are said to be orthogonal if
in balanced case \(\sum c_i d_i =0\)
\[ \begin cov(\hat_1, \hat_2) &= cov(\sum_i c_i \bar_, \sum_l d_l \bar_) \\ &= \sum_i \sum_l c_i d_l cov(\bar_,\bar_) \\ &= \sum_i c_i d_i \frac = 0 \end \]
Orthogonal contrasts can be used to further partition the model sum of squares. There are many sets of orthogonal contrasts and thus, many ways to partition the sum of squares.
A special set of orthogonal contrasts that are used when the levels of a factor can be assigned values on a metric scale are called orthogonal polynomials
Coefficients can be found for the special case of
- equal spaced levels (e.g., (0 15 30 45 60))
- equal sample sizes ( \(n_1 = n_2 = . = n_\) )
We can define the SS for a given contrast:
all contrasts have d.f = 1
21.1.3.2 Unbalanced
We could have unequal numbers of replications for all treatment combinations:
- Observational studies
- Dropouts in designed studies
- Larger sample sizes for inexpensive treatments
- Sample sizes to match population makeup.
Assume that each factor combination has at least 1 observation (no empty cells)
Consider the same model as:
\[ Y_ = \mu_ + \alpha_i + \beta_j + (\alpha \beta)_ + \epsilon_ \]
where sample sizes are: \(n_\) :
\[ \begin n_ &= \sum_j n_ \\ n_ &= \sum_i n_ \\ n_T &= \sum_i \sum_j n_ \end \]
Problem here is that
\[ SSTO \neq SSA + SSB + SSAB + SSE \]
(the design is non-orthogonal)
- For \(i = 1. a-1,\)
- For \(j=1. b-1\)
We can use these indicator variables as predictor variables and \(\mu_, \alpha_i ,\beta_j, (\alpha \beta)_\) as unknown parameters.
\[ Y = \mu_ + \sum_^ \alpha_i u_i + \sum_^ \beta_j v_j + \sum_^ \sum_^(\alpha \beta)_ u_i v_j + \epsilon \]
To test hypotheses, we use the extra sum of squares idea.
For interaction effects
\[ \begin &H_0: all (\alpha \beta)_ = 0 \\ &H_a: \text(\alpha \beta)_ =0 \end \]
\[ \begin &H_0: \beta_1 = \beta_2 = \beta_3 = 0 \\ &H_a: \text \beta_j = 0 \end \]
Analysis of Factor Means
(e.g., contrasts) is analogous to the balanced case, with modifications in the formulas for means and standard errors to account for unequal sample sizes.
Or , we can fit the cell means model and consider it from a regression perspective
If you have empty cells (i.e., some factor combinations have no observation), then the equivalent regression approach can’t be used. But you can still do partial analyses
21.1.4 Two-Way Random Effects ANOVA
\[ Y_ = \mu_ + \alpha_i + \beta_j + (\alpha \beta)_ + \epsilon_ \]
- \(\mu_<..>\) : constant
- \(\alpha_i \sim N(0,\sigma^2_), i = 1. a\) (independent)
- \(\beta_j \sim N(0,\sigma^2_), j = 1. b\) (independent)
- \((\alpha \beta)_ \sim N(0,\sigma^2_),i=1. a,j=1. b\) (independent)
- \(\epsilon_ \sim N(0,\sigma^2)\) (independent)
All \(\alpha_i, \beta_j, (\alpha \beta)_\) are pairwise independent
Theoretical means, variances, and covariances are
\[ \begin E(Y_) &= \mu_ \\ var(Y_) &= \sigma^2_Y= \sigma^2_\alpha + \sigma^2_\beta + \sigma^2_ + \sigma^2 \end \]
\(Y_ \sim N(\mu_,\sigma^2_\alpha + \sigma^2_\beta + \sigma^2_ + \sigma^2)\)
\[ \begin cov(Y_,Y_) &= \sigma^2_, j \neq j' \\ cov(Y_,Y_) &= \sigma^2_, i \neq i'\\ cov(Y_,Y_) &= \sigma^2_\alpha + \sigma^2_ + \sigma^2_, k \neq k' \\ cov(Y_,Y_) &= , i \neq i', j \neq j' \end \]
21.1.5 Two-Way Mixed Effects ANOVA
21.1.5.1 Balanced
One fixed factor, while other is random treatment levels, we have a mixed effects model or a mixed model
Restricted mixed model for 2-way ANOVA:
\[ Y_ = \mu_ + \alpha_i + \beta_j + (\alpha \beta)_ + \epsilon_ \]
- \(\mu_<..>\) : constant
- \(\alpha_i\) : fixed effects with constraints subject to restriction \(\sum \alpha_i = 0\)
- \(\beta_j \sim indep N(0,\sigma^2_\beta)\)
- \((\alpha \beta)_ \sim N(0,\frac\sigma^2_)\) subject to restriction \(\sum_i (\alpha \beta)_ = 0\) for all j, the variance here is written as the proportion for convenience; it makes the expected mean squares simpler (other assumed \(var((\alpha \beta)_= \sigma^2_\) )
- \(cov((\alpha \beta)_,(\alpha \beta)_) = - \frac\sigma^2_, i \neq i'\)
- \(\epsilon_\sim indepN(0,\sigma^2)\)
- \(\beta_j, (\alpha \beta)_, \epsilon_\) are pairwise independent
Two-way mixed models are written in an “unrestricted” form, with no restrictions on the interaction effects \((\alpha \beta)_\) , they are pairwise independent.
Let \(\beta^*, (\alpha \beta)^*_\) be the unrestricted random effects, and \((\bar)_^*\) the means averaged over the fixed factor for each level of random factor B.
Some consider the restricted model to be more general. but here we consider the restricted form.
Responses from the same random factor \((B)\) level are correlated
Hence, you can see that the only way you don’t have dependence in the \(Y\) is when they don’t share the same random effect.
An advantage of the restricted mixed model is that 2 observations from the same random factor b level can be positively or negatively correlated. In the unrestricted model, they can only be positively correlated.
Mean Square (A fixed, B random)
For fixed, random, and mixed models (balanced), the ANOVA table sums of squares calculations are identical. (also true for df and mean squares). The only difference is with the expected mean squares, thus the test statistics.
In Random ANOVA, we test
\[ \begin &H_0: \sigma^2 = 0 \\ &H_a: \sigma^2 > 0 \end \]
The same test statistic is used for mixed models, but in that case we are testing null hypothesis that all of the \(\alpha_i = 0\)
The test statistic different for the same null hypothesis under the fixed effects model.
Test for effects of (A fixed, B random)
Estimation Of Variance Components
In random and mixed effects models, we are interested in estimating the variance components
Variance component \(\sigma^2_\beta\) in the mixed ANOVA.which can be estimated with
Confidence intervals for variance components can be constructed (approximately) by using the Satterthwaite procedure or the MLS procedure (like the 1-way random effects)
Estimation of Fixed Effects in Mixed Models
Contrasts on the Fixed Effects
\[ \begin L &= \sum c_i \alpha_i \\ \sum c_i &= 0 \\ \hat &= \sum c_i \hat_i \\ \sigma^2(\hat) &= \sum c^2_i \sigma^2 (\hat_i) \\ s^2(\hat) &= \frac \sum c^2_i \end \]
Confidence intervals and tests can be constructed as usual
21.1.5.2 Unbalanced
For a mixed model with a = 2, b = 4
\[ \mathbf \sim N(\mathbf\beta, M) \]
where \(M\) is block diagonal
if we knew the variance components, we could use GLS:
but we usually don’t know the variance components \(\sigma^2, \sigma^2_\beta, \sigma^2_\) that make up \(M\)
Another way to get estimates is by Maximum likelihood estimationwe try to maximize its log
