Review of Underlying Concepts

• Variance & Covariance
  • Importance in statistical analyses
• Covariance & Correlation
  • Relationship between them
  • Why use one or the other?
  • Both are descriptive statistics
    • Even though tests can be run on them

Review of Underlying Concepts (cont.)

Review of Underlying Concepts (cont.)

• Partial & Semipartial Correlations
  • Semipartial correlations remove the effect of another variable from one of the correlated pair
  • Could remove the effect of several other variables from one of the pair
    • Which is conceptually the same thing we do in separating out the effects of various predictors on the same outcome

Review of Underlying Concepts (cont.)

• Correlations & Error
  • Correlations separate dispersion into variance & covariance
    • But make no assumptions about where error comes from
  • However, when testing significance of Pearson’s r, error is assumed to be normally distributed

Review of Underlying Concepts (end)

• Correlations & Error (cont.)
  • The (unshared) variances of both variables comprise the denominator
    • (This will be different for linear regression models)

Buildling the Equation

• Simplest form of a linear relationship is \(Y = bX\)
  • Where:
    • \(Y\) = Outcome / response / criterion / DV
    • \(X\) = Predictor / input / IV
    • \(b\) = Slope of \(X\)
      • The typical null hypothesis (H₀) of “no effect” is expressed here as:
        \(b\) = 0

Buildling the Equation (cont.)

\[Y = bX\]

Buildling the Equation (cont.)

Buildling the Equation (cont.)

\[Y = b_{0} + b_{1}X_{1}\]

Buildling the Equation (cont.)

\[Y = b_{0} + b_{1}X_{1}\]

• We are using the data to estimate:
  • \(b_{0}\): The beginning/baseline value (when \(X_{1}\) = 0)
  • \(b_{1}\): The effect of \(X_{1}\) on \(Y\)
    • How much a change in \(X_{1}\) corresponds to a change in \(Y\)

Buildling the Equation (cont.)

• The equation so far is the model: \(Y = b_{0} + b_{1}X_{1}\)
  • We are modeling a linear relationship between \(X_{1}\) and \(Y\)
  • Assuming linearity, \(b_{0}\) & \(b_{1}\) contain all of the information about that \(X_{1}\)-\(Y\) relationship
• We can test the significance and effect size of the whole model
  • How much of the relationship is accounted by that linear model

Buildling the Equation (cont.)

• But the model mis-estimates the actual data
  • There is error in the model’s estimations

Buildling the Equation (end)

More About the Equation

\[Y = b_{0} + b_{1}X_{1} + e\]

• Because error is separated out,
  • The value of \(X_{1}\) is assumed to be measured without error
• Separating the intercept, slope, & error means each can be estimated & modified separately
• Note that Kim, Mallory, & Valerio (2022) present this equation as: \(y = a + b \times x + \sigma\)

More About the Equation (cont.)

• Adding more specificity to the equation:
  \(\hat{Y}_{i} = b_{0} + b_{1}X_{i1} + e_{i}\)
  • \(\hat{Y}_{i}\) = Predicted value of \(Y\) for participant \(i\)
  • \(b_{1}\) = Slope for variable \(X_{1}\)
  • \(X_{i1}\) = Value on \(X_{1}\) for participant \(i\)
  • \(e_{i}\) = Error of measurement of participant \(i\)’s outcome

More About the Equation (end)

Linear Models vs. ANOVAs

Questions Best Addressed by ANOVAs vs. Linear Models

• ANOVAs (and ANCOVAs, MANOVAs, etc.) can ask:
  • Which variable is significant?
  • Is there an interaction between variables?
• Linear regressions can also ask:
  • What is the best combination of variables?
  • Does a given variable—or set of variables—significantly contribute to what we already know?

Signal-to-Noise in Linear Models

\(\hat{Y}_{i} = b_{0} + b_{1}X_{i1} ... + b_{k}X_{ik} + e_{i}\)

• The variance in \(\hat{Y}_{i}\) is divided into:
  • Changes due to the predictors—the “signals”
  • Changes due to “other things” (and relegated to error / noise term(s)
  • (N.b., the intercept, \(b_{0}\), is a constant when measuring \(\hat{Y}_{i}\), and so not included in this partitioning of the variance in \(\hat{Y}_{i}\))

Signal-to-Noise in Linear Models (cont.)

\(\hat{Y}_{i} = b_{0} + b_{1}X_{i1} ... + b_{k}X_{ik} + e_{i}\)

• The sum of squares representation of this partition into predictors & error looks like:

\[\sum\limits_{i=1}^{n} (Y_{i} - \overline{Y})^{2} = \sum\limits_{i} (\hat{Y}_{i} - \overline{Y})^{2} + \sum\limits_{i} ({Y}_{i} - \hat{Y}_{i})^{2}\]

Signal-to-Noise in Linear Models (cont.)

\(\sum\limits_{i=1}^{n} (Y_{i} - \overline{Y})^{2} = \sum\limits_{i} (\hat{Y}_{i} - \overline{Y})^{2} + \sum\limits_{i} ({Y}_{i} - \hat{Y}_{i})^{2}\)

• I.e.,the squared sum of the differences of each instance (\(Y_{i}\)) from the mean (\(\overline{Y}\)) equals:
  • The squared sum differences of each predicted value (\(\hat{Y}_{i}\)) from the mean
  • Plus the squared sums of differences of the actual values (\(Y_{i}\)s) from the respective predicted values

Signal-to-Noise in Linear Models (cont.)

• Another way of saying this:

\(\sum\limits_{i=1}^{n} (Y_{i} - \overline{Y})^{2} = \sum\limits_{i} (\hat{Y}_{i} - \overline{Y})^{2} + \sum\limits_{i} ({Y}_{i} - \hat{Y}_{i})^{2}\)

• Is to say this:
  \(\text{Total SS = SS from Regression + SS from Error}\)
  • Or, further condensed as:
    • \(SS_{Total} = SS_{Reg.} + SS_{Error}\)

Signal-to-Noise in Linear Models (end)

• Using \(SS_{Total} = SS_{Reg.} + SS_{Error}\),
  • We can compute the ratio of predicted to actual:
    Ratio of Predicted-to-Actual Variance = \(\frac{SS_{Reg.}}{SS_{Total}}\)
    • Or, equivalently as \(1 - \frac{SS_{Reg.}}{SS_{Error}}\)

Adding More Terms to Models

• Adding another variable to the equation:
  \(\hat{Y}_{i} = b_{0} + b_{1}X_{i1} + b_{2}X_{i2} + e_{i}\)
  • \(X_{i2}\) = Participant \(i\)’s value on the other variable (\(X_{2}\)) added to the model
  • \(b_{2}\) = Slope for \(X_{2}\)
• Since there are multiple predictors (\(X\)s) in this model,
  • This is called a multiple linear regression

More About the Equation (cont.)

• We can continue to add still more variables to the model, e.g., \(X_{3}\) and \(X_{3}\):

\[\hat{Y}_{i} = b_{0} + b_{1}X_{i1} + b_{2}X_{i2} + b_{3}X_{i3} + b_{4}X_{i4} + e_{i}\]

• When there are a lot of variables in the model, then we usually abbreviate the equation to:
  \[\hat{Y}_{i} = b_{0} + b_{1}X_{i1} ... + b_{k}X_{ik} + e_{i}\]
  • Were \(k\) is the number of variables

Adding More Terms to Models (cont.)

\(\hat{Y}_{i} = b_{0} + b_{1}X_{i1} ... + b_{k}X_{ik} + e_{i}\)

• We can test interactions by adding additional terms
  • E.g., \(... b_{1}X_{i1} + b_{2}X_{i2} + \mathbf{b_{3}(X_{i1}X_{i2}}) ...\)
• Or test non-linear effects, also by adding terms
  • E.g., \(... b_{1}X_{i1} + \mathbf{b_{2}X_{i1}^{2}} ...\)

Adding More Terms to Models (end)

• Just as we separated out the effects of the predictors,
  • We can separate out sources of error
    • E.g., per predictor/term in the model
• We can also combine error terms
  • E.g., we can “nest” one variable into another
    • Patients nested within hosptital units
    • Wave (time) nested within patient

Modeling Linear & Non-Linear Relationships

\(\hat{Y}_{i} = b_{0} + b_{1}X_{i1} ... + b_{k}X_{ik} + e_{i}\)

• \(Y\) is assumed to follow a certain distribution
  • This determines how error is modeled
  • E.g., is error usually assumed to be normally distributed
  • But both distributions can be assumed to be something else
    • E.g., logarithmic

Modeling Linear & Non-Linear Relationships (cont.)

\(\hat{Y}_{i} = b_{0} + b_{1}X_{i1} ... + b_{k}X_{ik} + e_{i}\):

• \(X\)s can be nominal, ordinal, interval, or ratio
  • This affects how those variables are modeled
  • As well as the error related to them
• We could transform the terms on the right
  • E.g., raise them to a power or take their log

Modeling Linear & Non-Linear Relationships (cont.)

Modeling Linear & Non-Linear Relationships (end)

Generalized Linear Models

Generalized Linear Models (cont.)

• N.b., confusingly, in addition to generalized linear models,
  • There are general linear models
  • “General linear model” simply refers to models you already know.
    • I.e., those with:
      • Normally-distributed, iid variables &
      • Identity link functions
    • Like ANOVAs & multiple linear regressions

Generalized Linear Models (cont.)

• Assumptions of generalized linear models:
  • Relationship between response and predictors must be expressible as a linear function
    • But many can model heteroscedasticity well
  • Cases must be independent of each other
    • Or that relationship should be part of the model
      • Often as nesting

Generalized Linear Models (end)

• Assumptions of generalized linear models (cont.):
  • Predictors should not be too inter-correlated (lack of multicollinearity)
    • Linear regression simply cannot get accurate measures of two effects if they cannot be easily separated
  • The random error & link functions should approximate their actual functions

Predicting BMI from Sex & Neighborhood Safety

• Predicting body mass index levels among adolescents from:
  1. Whether an adolescent is biologically female
  2. Whether they feel their neighborhood is safe
• via SPSS (v. 29 & 31)

Data Used

• From the National Longitudinal Study of Adolescent to Adult Health (Add Health)
  • Using the prepared add_health.sav dataset
  • Since data are longitudinal, only the first instance (wave) of data collection was used
    • Selected via:
      1. Data > Select Cases...
      2. Under If condition is satisfied, added Wave = 1 to select only the first wave

Descriptives

Descriptives (cont.)

Descriptives: Q-Q Plot

Descriptives: 2 × 2 Table

Correlations

Correlations with CIs

Linear Regression

• Conducted via:
  1. Analyze > Regression > Linear
  2. BMI as Dependent
  3. Bio_Sex & Feel_Safe_in_Nghbrhd as predictors in Block 1 of 1

Linear Regression (cont.)

Linear Regression (cont.)

Linear Regression (cont.)

Interpretting the Effects

• Writing these results in linear equation form:

Interpretting the Effects (cont.)

Interpretting the Effects (cont.)

Interpretting the Effects (cont.)

Multicollinearity

• When two or more predictors share too much variance
  • I.e., are strongly correlated
    • E.g., r ≥ .8 (Dormann et al., 2013)
• Two general sources:
  1. Structural: Caused by how the model was constructed
  • E.g., adding interaction terms
  1. Data: Caused by variables that are inherently correlated

Multicollinearity (cont.)

• Problems caused by multicollinearity:
  • Parameter estimates of multicollinear terms can be unstable
    • And even reverse sign
  • Reduces the power of the whole model
    • Because the parameter estimates are less precise

Multicollinearity (cont.)

• Multicollinearity doesn’t typically affect the whole model’s R²
  • Or the model’s good-of-fit statistics
• It mostly impairs interpretation of individual predictors
• Can be tested with variance inflation factor (VIF)
  • VIF ranges from 1 to \(\infty\)
  • Where values >10 may indicate problems (e.g., Salmerón et al. 2016)

Multicollinearity (cont.)

• Addressing multicollinearity
  • Centering variables (subtracting the mean) can reduce structural multicollinearity (Iacobucci et al., 2016)
  • Remove one of the correlated variables
  • Only test/compare overall model fit
  • Use another analysis
    • E.g., canonical correlations or principal component analysis

Multicollinearity (end)

• Multicollinearity is typically not a concern if the variables with high multicollinearity are:
  • Control variables
  • Intentional products of other variables
    • E.g., interaction terms, raised to a power, etc.
  • Dummy variables

Independence of Cases

• When one case (participant, round of tests, etc.) is correlated with another case
• Can also produce unstable parameter estimates
  • Thus affecting significance tests
    • Through both false positives (Type 1) & false negatives (Type 2)
• May also affect model goodness of fit
  • And not isolated to a few predictors

Independence of Cases (cont.)

• Addressing non-independence
  • Best is through research design
  • Can also model inter-dependence
    • E.g., nesting cases
      • As is done explicitly in multilevel (hierarchical) models