Logistic Regression

Overview

Problem of Linear Regression with Dichotomous Outcomes
From Yes or No to Probability, Odds, and Log Odds
A Logistic Regression Example

Problem of Linear Regression with Dichotomous Outcomes

Dichotomous Outcomes Are Not Linear

Binary observations with an illustrative straight line extending outside zero to one and a logistic curve remaining within these bounds.

E.g., Difficulty Standing From a Chair & BMI

Observed chair-difficulty responses at zero and one, with linear and logistic regression fits using measured BMI as the sole predictor.

At high BMI values, the linear fit exceeds 1. The logistic fit stays within bounds.

HRS data, n = 477. Chair difficulty (R16CHAIR): 0 = No, 1 = Yes.

Unequal Variance Appears in the Residuals

Chair-difficulty observations at zero and one, with vertical lines to their fitted values from linear regression on measured BMI. Darker blue indicates larger absolute residuals.

Remember that linear regression assume homogeneity of variance
The mis-estimate of each outcome varies systematically across the range

HRS data, n = 477. Linear regression on measured BMI (R16PMBMI).

The vertical segments are raw residuals from the same unadjusted linear regression of chair difficulty on measured BMI shown on the earlier BMI slide. The observed outcome remains zero or one. Both models use the 477 participants with complete chair-difficulty and measured BMI values. Points use the original BMI values, so some overlap. Each residual line joins an observation to the fitted line at its actual BMI value.

Color shows absolute residual size, not a variance estimate. Compare the two residual bands and use the preceding theoretical curve to explain why binary variance depends on event probability. The linear model predicts above one at the upper end of the observed BMI range. Those predictions are impossible probabilities and illustrate a separate limitation of linear probability models.

Binary Outcomes Have Unequal Variance

Binary outcomes vary most when Yes & No are equally likely.

The conditional variance p times one minus p is 0.09 at probabilities 0.10 and 0.90 and reaches 0.25 at probability 0.50.

From Yes / No to a Logistic Model

Logistic Regression Changes the Model

In logistic regression, each person still has a Yes or No outcome
But we use the data to estimate how the probability of Yes varies with the predictors
- We then convert that probability to odds (and then log odds)
- For reasons I’ll explain next
Best is to use an example…

1. Start With Individual Responses

Among the 500 people in our Wave 16 HRS sample:
- 263 reported difficulty getting up from a chair
- 237 reported not having such difficulty

Response	Recorded Value	Number of People
Yes	1	263
No	0	237

Note that each person’s response is still either 0 or 1.

2. Describe the Probability of Yes

Probability compares the people who answered Yes with everyone.

There are 263 Yes responses among these 500 people
Dividing 263 by 500 = 0.526 (or 52.6%)
- I.e., slightly more than half report difficulty getting out of a chair

Probability answers: “What proportion of the whole group said Yes?”

3. Change the Comparison to Odds

Odds compare the people who answered Yes with those who answered No.

Keep the same 263 Yes responses
Compare them with the 237 No responses, rather than all 500 people
Dividing 263 by 237 = the odds are approximately 1.11

The data haven’t changed; just the denominator has.

Converting Probability to Odds

Equivalently, we could convert probability to odds

Here, the probability of Yes is 52.6%
To convert to odds:

Find the probability of No: (1 – .536 =) 47.4%.
Divide the probability of Yes by the probability of No:
- .526 / .474 ≈ 1.11

This gives the same answer as comparing 263 Yes with 237 No.

4. Change Odds to Log Odds

The software takes the natural logarithm of the odds.

ln(1.11) = 0.104

Probability	Odds	Log Odds
52.6%	1.11	0.104

This log odds is also called a “logit”
These are the values we estimate
- But usually convert back to odds (or probabilities) in the output

Why Use Log Odds?

Probabilities must stay between 0 and 1.
Odds can rise above 1, but they cannot be negative.
Log odds can be negative, zero, or positive, without a fixed boundary.
This lets the model add predictor contributions on an unrestricted scale.

From Odds to Log Odds

Odds	Log Odds (Logit)
0.10	−2.30
0.20	−1.61
0.25	−1.38
0.33	−1.11
0.50	−0.69
1.00	0.00
2.00	0.69
3.00	1.10
4.00	1.38
5.00	1.61
10.00	2.30

How Odds Become Log Odds

An increasing, flattening curve maps odds to their natural logarithm. Marked odds are 0.10, 0.20, 0.25, 0.33, 0.50, 1, 2, 3, 4, 5, and 10. Each dot is labeled with its logit value. Odds below one have negative log odds; odds of one have log odds zero; odds above one have positive log odds.

Equal Steps Can Mean Multiplying the Odds

Odds	Log Odds	Absolute Value of Change in Odds
0.125	−2.07	0.69
0.25	−1.38	0.69
0.50	−0.69	0.69
1.00	0.00	0.69
2.00	0.69	0.69
4.00	1.38	0.69
8.00	2.07	0.69

Each doubling/halving of the odds changes the log odds by 0.69

5. Build the Regression on This Scale

The model begins with a starting log-odds value
It adds a contribution for each predictor, such as BMI or education
A positive contribution raises the modeled odds; a negative one lowers them
The software estimates these contributions from the observed Yes and No responses

6. Return to an Understandable Probability

For our running example, reversing the steps gives:

Step	Result
Begin with the log odds	About −1.39
Undo the logarithm	Odds of 0.25: 1 Yes for every 4 No
Count Yes as a share of everyone	1 Yes among 5 people
Express that share as a probability	20%

The Responses Stay Binary Throughout

What We Observe	What the Model Estimates
Each person’s `Yes` or `No`	Probability of Yes for their predictor values
Recorded as 1 or 0	Built on the log-odds scale
Used to estimate the model	Converted back to a probability for interpretation

N.b., we do not take the logarithm of each person’s 0 or 1.

From a Coefficient to an Odds Ratio

Software can express a predictor’s contribution as an odds ratio.

An odds ratio of 1.00 means unchanged odds
An odds ratio of 1.50 means 50% higher odds per predictor unit
An odds ratio of 0.75 means 25% lower odds per predictor unit

(Other modeled predictors are held constant.)

Doubling the Odds Does Not Double Probability

Starting probability	Starting odds	Doubled odds	New probability
.10	.111	.222	.182
.50	1.000	2.000	.667

A fixed odds ratio can correspond to different probability differences.

The Curve Flattens Near the Boundaries

Near .50, a change in a predictor can move probability substantially.
Near 0 or 1, the same predictor change moves probability less.
As the curve approaches either boundary, it becomes flatter.

A constant effect on the odds produces a changing effect on probability.

The Same Odds Effect, Different Probability Changes

Suppose each one-unit predictor increase raises the odds by 50%.

Starting Probability	New Probability	Increase
5%	7.3%	2.3 percentage points
50%	60.0%	10.0 percentage points
95%	96.6%	1.6 percentage points

The odds effect is identical in all three rows.

Illustrative probabilities, with other predictors held constant.

The Same Probability Increase Requires More Change

To raise probability by 10 percentage points:

Probability Change	Required Predictor Increase
50% to 60%	1 unit
5% to 15%	About 3 units
85% to 95%	About 3 units

Moving the same vertical distance takes more horizontal movement near the ends of the curve.

Same illustrative model: a 50% increase in odds per predictor unit.

Estimation and Statistical Tests

Maximum likelihood finds coefficients that best support the observed outcomes.
Likelihood-ratio tests compare nested models.
Wald tests assess coefficients relative to their standard errors.
Confidence intervals describe precision as well as direction.

Assumptions and Checks That Matter

Independent observations, or a model that accounts for clustering.
Appropriate predictor forms on the log-odds scale.
No perfect separation or severe multicollinearity.
Adequate events and nonevents for the parameters estimated.

An SPSS Analysis, Step by Step

1. Define the Event and the Purpose

Question: Which characteristics are associated with difficulty getting up from a chair?

SPSS outcome: R16CHAIR.
- Coding: No = 0, Yes = 1.
Predictors:
- Education
- BMI
- Neuroticism
- Gender

2. Account for the Analysis Sample

Case Processing Summary	Cases	Percent
Included	469	93.8%
Missing	31	6.2%
Total	500	100.0%

What information might differ between included and excluded participants?

3. Verify Coding and Measurement

Predictor	How SPSS modeled it
Gender	`Male` = `1`, `Female` = `2` (Based on HRS’s coding scheme.)
Highest educational degree	One numeric coefficient
BMI and Neuroticism	One coefficient per scale unit

Note one-unit changes mean very different things here.

4. Fit the Planned Model

Analyze > Regression > Binary Logistic
Specify categorical predictors and their reference categories
Use Enter for a prespecified predictor set
Request 95% confidence intervals for Exp(B)

5. Assess the Model as a Whole

The overall model test gives χ² = 89.95, with 4 dfs and p < .001.

The predictors jointly improve fit over the intercept-only model.
Four coefficients contribute to this comparison.
Statistical improvement does not establish useful prediction.

Attached output: Omnibus Tests of Model Coefficients.

6. Interpret Adjusted Associations

Predictor	Adjusted OR	95% CI	p
Education, per code unit	0.77	0.69 – 0.86	< .001
BMI, per unit	1.10	1.06 – 1.13	< .001
Neuroticism, per unit	2.05	1.49 – 2.82	< .001
Female vs. male	1.15	0.76 – 1.74	.516

ORs from Variables in the Equation. Wald confidence intervals calculated from the stored coefficients and standard errors.

An Odds Ratio in Words

BMI: Each one-unit increase corresponds to about 9.8% higher odds of difficulty, adjusting for the other predictors.

For a five-unit difference:
- Apply the per-unit odds multiplier five times: the odds ratio is about 1.60
- I.e., about 60% higher odds, assuming the same odds multiplier for each additional BMI unit

An Imprecise Association

Female vs. male: adjusted OR = 1.15, 95% CI [0.76, 1.74], p = .516.

The interval includes lower and higher odds
The data do not clearly distinguish this adjusted association from no association

7. Separate Fit From Calibration

Output	Result	Interpretation
Cox-Snell pseudo-R²	.175	An index of model fit. Summarizes improvement over the intercept-only model; its maximum is below 1.
Nagelkerke pseudo-R²	.233	An index of model fit. Rescales Cox-Snell to span between 0 & 1
Hosmer–Lemeshow	χ² = 7.38 df = 8 p = .497	No signigicant model misfit to the data

8. Read the Classification Table

Outcome Level	Observed*	Predicted `No`	Predicted `Yes`
`No`	228	148	80
`Yes`	241	77	164

* After missing data

Specificity: 148 of 228 without difficulty were identified: 64.9%.
Sensitivity: 164 of 241 participants with difficulty were identified: 68.0%.
Accuracy: 312 of 469 participants were classified correctly: 66.5%.

Specificity: The proportion of nonevents correctly classified No.
Sensitivity: The proportion of events correctly classified Yes.
Accuracy: Total proportion of events & nonevents correctly classified.

A Baseline and a Threshold Matter

Predicting Yes for everyone is already 51.4% accurate
The fitted model reaches 66.5%, a 15.1-point increase
A .50 threshold may not match the consequences of errors
These are results in the same sample used to fit the model.

9. Investigate Model Weaknesses

Evaluate nonlinear effects, especially BMI and education coding.
Examine sparse categories, collinearity, and influence.
Case 394: observed No, predicted probability .884 (88.4% change of being Yes).
Investigate unusual cases before considering exclusion.

A Concise Results Statement

Among 469 complete cases, the model improved fit over the intercept-only model, χ² = 89.95, df = 4, p < .001. Adjusting for education, Neuroticism, and gender, each one-unit increase in BMI was associated with higher odds of difficulty getting up from a chair (OR = 1.10, 95% CI [1.06, 1.13]).

At a .50 cutoff, sensitivity was 68.0% and specificity was 64.9% in the analysis sample. Predictive performance requires validation.

The Analysis Workflow

Define the event, question, predictors, and reference groups.
Inspect coding, missingness, sample size, and dependence.
Fit the planned model and check its specification.
Interpret adjusted odds ratios with confidence intervals.
Evaluate and report performance appropriate to the goal.