What ANOVAs Test

• ANOVAs are designed to test if two or more groups differ in the mean value of the outcome
  • E.g., if patients of different races tend to have different mean levels of blood lipids (e.g., LDLs)
• So:
  • The outcome variable (DV) is continuous
  • The main predictors (IVs) are categorical
  • More complex models can test more complex & specific types of relationships

What ANOVAs Test (cont.)

• ANOVAs are also designed to conduct “omnibus” tests
  • If a predictor is found to be significant, it means that there is one or more significant difference somewhere in that variable
    • E.g., that one or more racial groups have different mean levels of blood LDLs

What ANOVAs Test (end)

• The use of an initial, omnibus is intentional:
  • The initial omnibus test is used to control for conducting too many significance tests
  • And thus helping control for false positives (Type 1 errors)
• Remember:
  • Test of one variable alone is called a main effect
  • Test between two (or more) variables are called interactions

Examples of What ANOVAs Test

Examples of What ANOVAs Test (cont.)

• Effect of communication training rubric on empathic communication skills of nurses at a tertiary care hospital (Rawat et al., 2024)
  • “After the intervention, all measurements conducted on experimental groups at various locations yielded better results”
• Effects of single- and double-shift work on hand and cognitive functions in nurses (Ulupinar, 2024)
  • “[T]he double-shift work significantly exacerbated declines in all measured functions.”

ANOVAs as Semipartial Correlations

ANOVAs as Semipartial Correlations (cont.)

Assumptions of ANOVAs

• The “ANOVA family” is a group of similar tests
• That all follow the basic idea of a linear regression
  • I.e., assuming the relationship between the predictors (IVs) and outcome (DV) is linear
  • And assuming any deviations from linearity are entirely due to error
• Also assumes the variables and error are all normally distributed
  • Since it uses F-tests, this assumption is worth investigating more than usual

Factors Associated with End-of-Day Fatigue

* p < .05

Main Effects & Interactions Example (cont.)

Main Effects & Interactions Example (cont.)

Main Effects & Interactions Example (end)

Roles of t- & F- Tests in ANOVAs

• Remember, the main use of ANOVAs is to test:
  • Differences between groups
  • In the levels of an outcome variable
• E.g.:
  • Differences in those with and without COPD
  • In their levels of cognitive functioning
• Or:
  • Differences between those with COPD, asthma, & bronchiectasis
  • In their levels of cognitive functioning

Roles of t- & F- Tests in ANOVAs (cont.)

• To test group differences, ANOVAs use t and F tests:

t & F as Tests of Mean Differences

• t- and F-tests are tests of mean differences
  • t-tests are used for two means
  • F-tests are used for three or more means
• I.e., they are used to test whether two means are significantly different from each other
  • They can also be used to test if means are different than zero

t & F as Tests of Signal-to-Noise Ratio

• Generally, information in a sample of data is placed into two categories:
  • “Signal,” e.g.:
    • Difference between group means,
    • Magnitude of change over time, or
    • Amount two variables co-vary/co-relate
  • “Noise,” e.g.,
    • Differences within a group
    • “Error”—anything not directly measured

As Signal-to-Noise Ratio (cont.)

• E.g., the F-test in an ANOVA
  • A ratio of the “mean square variance” between an effect’s groups/levels
    • Versus the “mean square error” within each group
  • If F < 1, then there is more noise than signal
    • And no chance off significance
  • If F > 1, then use sample size to determine if the value is big enough to be significant

Example of Signal-to-Noise Ratio

• \(\frac{\text{Signal}}{\text{Noise}}=\frac{\text{Mean Square Effect}}{\text{Mean Square Error}}\)
• E.g., for the between groups effect: \(\frac{134.613}{2.159} = 62.360\)

The ANOVA Family

• There is a “family” of tests that all are related mathematically
  • And all given names similar to “ANOVA”
• All use ordinary least squares to compute the linear regression
  • And similar (but not identical) methods to estimate error
  • Differences in computing the error term is the main way they differ

The ANOVA Family (cont.)

• Common assumptions of all ANOVA-family models
  • Normally distributed errors
  • Homogeneity of variance (except in robust methods)
• All require conducted post hoc analyses to find specific differences between levels of significant categorical predictors
  • E.g., if there is a significant main effect for (non-binary) gender,
    • We must then conduct post hocs to see which genders had different outcomes

Types of ANOVAs

One-Way ANOVA

• Compares means across 3+ levels of one predictor (IV)
  • (Can be used to test between 2 levels, but t-tests are better)
• Assumes:
  • Normality (technically of “residuals”)
  • Homogeneity of variances
  • Independent observations

One-Way ANOVA Example

Kusi Amponsah, A., Oduro, E., Bam, V., Kyei-Dompim, J., Ahoto, C. K., & Axelin, A. (2019). Nursing students and nurses’ knowledge and attitudes regarding children’s pain: A comparative cross-sectional study. PloS One, 14(10), e0223730-. doi: 10.1371/journal.pone.0223730

Two-Way ANOVA

• Tests effects of two categorical IVs
  • (On one continuous DV)
• Can evaluate both:
  • Main effects &
  • Interaction
• Assumptions similar to one-way ANOVA
• E.g., Arsat et al. (2023): Effects nurse experience & work setting on caring behavior

ANCOVA

• Includes both categorical & continuous predictors (IVs)
  • Continuous are added to control for their effects
  • But they are often investigated for significance, interactions, etc.
• E.g., Solera-Sanchez et al. (2021): Effects of physical activity, diet adherence, sleep quality, sleep duration, and screen time on health-related quality of life in adolescents

Repeated Measures ANOVA

• Used when same participants measured more than once
• Accounts for within-participant correlations
• Assumptions:
  • “Sphericity,” that scores don’t get more (or less) spread out over time
  • And normality, etc.
• E.g., Gattinger et al. (2023): Effects of an educational program on nurses’ competence, self-efficacy, and musculoskeletal health

MANOVA

• “Multivariate ANOVA”: multiple DVs analyzed simultaneously
  • Thus tests whether groups differ on a combination of outcomes
• Requires multivariate normality
• E.g., Shaygan et al. (2022): Assessed changes in pain intensity and pain catastrophizing between the two groups over time
  • A repeated-measures MANOVA: Alzoubi et al. (2023)

Mixed-Design ANOVA

• Mixed design combines within- and between-participants factors
  • An other way to test effect over time
• Evaluates:
  • Time effects (within)
  • Group differences (between)
  • Interaction
• E.g., Erwin et al. (2016): A two-way mixed-design ANOVA was employed with time of testing (morning, afternoon, evening) as the within-participants factor and gender as the between-participants factor

Repeated Measures vs. Mixed-Design

• Repeated measures ANOVA
  • All independent variable levels applied to the same participants
    
  • “Do participants change across time or conditions?”
• Mixed-design ANOVA
  • Some factors vary within participants; others vary between participants
    
  • “Do groups differ, do individuals change, and do these effects interact?”

Repeated Measures vs. Mixed-Design (cont.)

• Use repeated measures when all data are within-participant
  
• Use mixed design when combining within- and between-participant comparisons

Repeated Measures vs. Mixed-Design (end)

Post hoc Analyses

Post hoc Analyses (cont.)

• General strategy is to set the \(\alpha\) so that it stays at 0.05 for the group of tests
  • If conducting multiple comparisons, adjustments must be made to control the family-wise error rate (FWER)
    
  • Without adjustments, the probability of making at least one false positive increases as the number of comparisons increases

Post hoc Analyses (end)

• E.g., assume we’re making comparing three groups / levels of a predictor in post hoc analyses:
  • Each test uses \(\alpha\) = .05
    
  • Probability of at least one false positive:
    P(at least one error) = 1 - (1 - .05)³
                = 1 - (.95)³ = 1 - .857 = .143
• Can thus adjust \(\alpha\) per test to be \(\frac{.05}{3}\) = .0167
  • Ensures overall FWER does not exceed .05
• This is called the Bonferroni correction
  • It is simple, but under-powered (Perneger, 1998)

Summary of Common Post Hocs

When to Use Which Post Hoc

Planned Comparisons

• Instead of using an F-test followed by all possible comparisons in post hoc analyses,
  • Can create a set of planned comparisons of only a subset of interesting tests
    • Including of interaction effects
  • Can still use family-wise error correction if desired
• Theory-driven & flexible
• Doesn’t waste power on unnecessary comparisons
• But less open to serendipity

Key Terms & Concepts

More Terms & Concepts