What Hypothesis Is Being Tested

• Research questions are often couched in terms of an “alternate” hypothesis (H_A) that there is some difference
  • E.g., that a mean is different than some value (e.g., the mean is not zero)
  • Or that there is a difference between two means
• (Ignore for now that other times we’re testing not a difference, but a relationship)

What Hypothesis Is Being Tested (cont.)

• But we actually test the viability of a “null” hypothesis (H₀)
  • E.g., the probability of finding our sample mean if the population mean is actually zero
  • Or the probability of finding our mean difference if there really is no true difference between the groups
• So, the p-value we report is the probability of finding our results if the null hypothesis is true

E.g., Testing Against H₀ = 0

Testing Against an
Other H₀ Distribution

Testing Between Two Distributions

Couched as Signal & Noise

1. Increase the Signal

Increasing the Signal

• We can best increase the signal through good research design:
  • Improve the effectiveness of our intervention
  • Investigate impactful things
  • Measuring—and thus being able to detect—a wide range of outcomes
    • Including among a wide range of participants/situations

2. Reduce the Noise

Reducing the Noise

1. Improve measurement precision
   • Reduce the noise from imprecise measurements
     • More reliable instruments/measurements
   • Make finer measurements—not just more accurate
2. Improve sampling techniques
   • Controlled data collection situations
   • Careful use of instructions, ordering of items/instruments
   • Consideration of sources of bias
     • And reasons for missing data / refusal to answer—or even participate

Reducing the Noise (cont.)

Reducing the Noise (end)

Understanding Power

• Conceptually, power is our ability to see things clearly
  • A lot like increasing the magnification of a microscope

Understanding Power (cont.)

Understanding Power (cont.)

• In hypothesis testing, a lack of power is the probability of getting a false negative
  • I.e., the chance of missing an effect that actually is there
• High power means little chance of missing a real effect
  • Low power means a real effect can easily be missed
• The probability of a false negative is written as β
  • (To correspond with the chance of a false positive, which is written as α)
  • So power—i.e., not missing a real effect—is
    1 – β

Understanding Power (cont.)

Understanding Power (end)

• Yes, a false positive (α) is also called a Type 1 error
• And a false negative (β) is also called a Type 2 error
  • But everyone gets confused by that, so I just stick with “false positive” & “false negative”

Increasing Power

How can we increase power?

1. Increase signal (also called “effect size”)
2. Reduce noise
   • Through good measurement, sampling & design, and accounting for unexplained variance
   • By increasing sample size

Changing Power by Changing α

Changing Power by Changing α (cont.)

• One-tailed tests are more powerful than two-tailed tests
• But ask a more specific—restrictive—question:
  • Two-tailed tests ask:
    “Is there any difference between these groups?”
  • One-tailed tests ask:
    “Is Sample 2 larger than Sample 1?”

Changing Power by Changing α (end)

• We generally use two-tailed tests
  • They are more conservative
  • Allow us to see even unexpected effects

Post hoc Power Analysis

• Aka “observed” (or “retrospective”) power
• Remember:
  • When conducting hypothesis tests, we are actually assuming the null hypothesis (H₀) is true
    • E.g., α is the chance of a false positive—incorrectly assuming a sample value is not from H₀
    • While all the time still assuming that null hypothesis is true

Post hoc Power Analysis (end)

• Power, then, is the chance of finding an effect
  • Assuming the null hypothesis is true
• If we find significance, though, we know no longer assume the null is true
  • So conducting power analyses after hypothesis testing uses refuted assumptions
• Post hoc power is thus nearly always fallacious
  • Post hoc power is only justified when the population values are know (e.g., O’Keefe, 2007)
  • And thus can be computed based on that distribution(s) instead of the null

Definition

“‘[T]he degree to which the phenomenon is present in the population’ or ‘the degree to which the null hypothesis is false.’” (Cohen, 1988, pp. 9 – 10)

• The size of a difference
  • Between groups, a different times, etc.
  • Between the odds/risks of various groups, etc.
    • E.g., higher odds ratio
• The strength of relationships between variables
  • Higher correlations
  • The tendency for things to co-occur
    • E.g., higher odds

Effect Size is the “Signal”

Effect Size Statistics

• Effect size statistics are standardized
  • z scores
    • I.e., SDs
  • Proportions
    • Correlations (proportions of variance accounted for)
    • Odds ratios, risk/hazard ratios
• We can thus compare a given effect size statistic across similar studies
  • (This is what is done in meta-analyses)
• However, we cannot directly compare two different effect size statistics
  • But we can convert them mathematically

Effect Size Statistics (cont.)

• We can also present effect sizes with, e.g., 95% confidence intervals
  • For differences and relationships,
    • The effect is significant if the 95% CI doesn’t overlap zero
  • For odds, risks, hazards (& their ratios)
    • It’s significant if the 95% CI doesn’t overlap 1
• This can complement formal significance tests
  • And also shows not only the size, but our, well, confidence in any conclusions made about it

Common Effect Size Statistics

• More info, including a few less common effect size statistics, is in the Companion

Those Deuced Sizes

• I.e., whether an effect size statistic is “small,” “medium,” or “large”
• Cohen proposed values for each measure he reivewed/created
• Cohen never meant his criteria to be canonical
  • But to serve “as a convention” while pointing out that “all conventions are arbitrary” (p. 13)
    • “The terms ‘small,’ ‘medium,’ and ‘large’ are relative, not only to each other, but to the area of behavioral science or even more particularly to the specific content and research method being employed in any given investigation” (p. 25)

Those Deuced Sizes (cont.)

• “A reader who finds that what is here defined as ‘large’ is too small (or too large) to meet what [their] area of behavioral science would consider appropriate standards is urged to make more suitable operational definitions. (p. 79)

A “Small” Effect

A “Small” Effect (cont.)

A “Small” Effect (cont.)

A “Small” Effect (end)

• About 85% overlap between two distributions
• 1% of total variance accounted for (2% for R²)
• Detecting prevalences of 55:45 versus 50:50
• E.g.:
  • Mean difference in IQ between twins and non-twins
  • Approximately the size of the difference in mean height between 15- and 16-year-old girls (i.e., 0.5 in. where the σ \(\approx\) 2.1, p. 26)
• For significance, would expect to need¹ Ns of:
  • t test: 394
  • r: 782

A “Medium” Effect

A “Medium” Effect (cont.)

A “Medium” Effect (cont.)

A “Medium” Effect (end)

• ~67% overlap between two distributions
• ~10% of total variance accounted for (13% for R²)
• Detecting prevalences of 65:35 versus 50:50
• E.g., a mean difference in IQ between members of clerical and semi-skilled occupations
  • Correlation between adolescents’ creativity & IQ
  • Approximately the size of the difference in mean height between 14- and 18-year-old girls (i.e., 1 in. where the σ = 2)
• For significance, would expect to need Ns of:
  • t test: 64
  • r: 84

A “Large” Effect

A “Large” Effect (cont.)

A “Large” Effect (cont.)

A “Large” Effect (end)

• ~53% overlap between two distributions
• 25% of total variance accounted for (~26% for R²)
• Detecting prevalences of 75:25 versus 50:50
• E.g., a mean difference in IQ between “between holders of the Ph.D. degree and typical college freshmen”
  • Or “between college graduates and persons with only a 50-50 chance of passing in an academic high school curriculum” p. 27)
• For significance, would expect to need Ns of:
  • t test: 26
  • r: 28

Why Use Effect Size?

• “Being forced to think in more exact terms than demanded by the Fisherian alternative ([i.e., that an effect size] is any nonzero value) is likely to prove salutary.
  
  [Researchers] can call upon theory for some help in answering the question and on [their] critical assessment of prior research in the area for further help [in using effect size to make nuanced decisions about data].” (p. 12)