Classification: Logistic Regression, Discriminant Analysis and the ROC Curve

Linear Regression, Read Correctly predicted a number. Many questions ask for a category instead: will a customer default, is an email spam, which of three diseases does a patient have. This post covers the classical classifiers for such questions: logistic regression, which models the probability of a class directly, and discriminant analysis (LDA, QDA, naive Bayes), which models how each class generates its data and applies Bayes’ theorem. Then it turns to measuring classifiers, where a low error rate can describe a useless one.

All four methods are implemented from scratch for this post (logistic regression by Newton’s method, the others in closed form), on simulated data where the truth is known, and the lab runs the same code in the browser.

1. Why Not Linear Regression?

Coding the classes as numbers and running least squares is always possible, but rarely sensible.

With three or more classes, the coding invents an order and equal distances: coding stroke, overdose and seizure as 1, 2, 3 claims that an overdose lies between the other two, and a different coding gives a different model. Only for ordered categories with roughly equal steps can this make sense.

With two classes, coded 0 and 1, it is less absurd. The fitted values estimate $Pr (Y = 1 ∣ X)$ in a crude way, but nothing keeps them inside [0, 1]. On simulated credit data (10,000 customers, 5.1% default; §3), the least-squares line of default against card balance gave 28.5% of customers a negative “probability” (down to −0.125).

A less obvious fact: for two classes, the least-squares coefficients point in exactly the same direction as linear discriminant analysis (§6): the cosine between the two coefficient vectors was 1.000000 in the experiment (Hastie et al., 2009, §4.3). What differs is the cut-off: with unequal class sizes (80% and 20%), thresholding the fitted values at 0.5 gave a test error of 0.158, against 0.154 for LDA’s own boundary. So “linear regression on a binary outcome is the same as LDA” is true for the direction, not for the classifier.

2. Logistic Regression

Logistic regression models the probability of class 1 with the logistic (sigmoid) function of a linear predictor (Berkson, 1944; Cox, 1958):

p (X) = Pr (Y = 1 ∣ X) = \frac{e ^{β_{0} + β_{1} X}}{1 + e ^{β_{0} + β_{1} X}} = \frac{1}{1 + e ^{- (β_{0} + β_{1} X)}} .

The function $σ (z) = 1/ (1 + e^{- z})$ maps every real number to the open interval (0, 1): it approaches 0 and 1 but never reaches them, and $σ (0) = 0.5$ exactly, so the curve crosses 0.5 where $β_{0} + β_{1} X = 0$ . Rearranged, the model is linear in the log-odds:

lo g \frac{p ( X )}{1 - p ( X )} = β_{0} + β_{1} X .

Reading a coefficient. $β_{1}$ is the change in the log-odds per unit of $X$ , so a unit increase multiplies the odds $p / (1 - p)$ by $e^{β_{1}}$ , the odds ratio. It does not change $p$ by a fixed amount: the same step in $X$ moves $p$ a lot near 0.5 and little near 0 or 1. $β_{1} > 0$ means higher $X$ , higher probability; $β_{1} = 0$ means no relationship.

Fitting by maximum likelihood. The coefficients are chosen to make the observed outcomes as probable as possible, maximising

ℓ (β_{0}, β_{1}) = i : y_{i} = 1 \prod p (x_{i}) i : y_{i} = 0 \prod (1 - p (x_{i})),

usually by Newton’s method, which for this model is a sequence of weighted least-squares fits (iteratively reweighted least squares). On the credit data it converged in 9 iterations:

default ~ balance	Estimate (SE)	Meaning
Intercept	−9.998 (0.292)	log-odds of default at balance 0
Balance ($)	0.00531 (0.00018)	odds × 1.70 per extra $100 (e^0.531)
Predicted $Pr (default)$	0.0091 at $1, 000; * * 0.650 * * a t$ 2,000	the first $1,000 adds 0.9 points, the second 64

With several predictors, the log-odds are linear in all of them, $lo g \frac{p}{1 - p} = β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p}$ , and each coefficient is read with the others held fixed, with the same caveats as in linear regression.

3. Confounding: Two Right Answers

The simulated credit data contain the confounding of Describing Data: students carry higher balances ( $1, 250 a g ain s t$ 807 on average), and higher balances raise default; but at any given balance, students default less.

Model	Student coefficient	Odds ratio
default ~ student	+1.260 (z = 13.6)	3.53
default ~ balance + student	−0.827 (z = −6.6)	0.44
At a balance of $1,500	P(default) 0.081 for a student	0.169 for a non-student

Both answers are correct, to different questions. A bank that knows nothing but student status should treat students as riskier (they default at 10.0% against 3.1%); among customers with the same balance, students are the safer group. A single-predictor model hides the second fact, and the reversal is a warning against reading any coefficient without asking what else it is conditioned on.

4. More Than Two Classes

For $K$ classes, multinomial logistic regression gives each class $k$ a linear function and normalises with the softmax:

Pr (Y = k ∣ X = x) = \frac{e ^{β_{k 0} + β_{k}^{⊤} x}}{\sum _{l = 1}^{K} e ^{β_{l 0} + β_{l}^{⊤} x}} .

Adding the same vector to every class’s coefficients leaves all probabilities unchanged, so only $K - 1$ of the functions are identifiable; one class is fixed as a baseline (or the coefficients are constrained to sum to zero). The softmax output layer of a neural network classifier is exactly this model applied to learned features.

5. When the Classes Separate: No Maximum Likelihood

A common statement is that logistic regression becomes “unstable” when the classes are well separated. It is more drastic than that. If some line separates the two classes perfectly, the likelihood can always be increased by making the sigmoid steeper, so the maximum-likelihood estimate does not exist (Albert & Anderson, 1984): the coefficients grow without bound. On 30 separable points:

Newton iteration	Slope	Log-likelihood
1	2.73	−7.43
10	42.85	−0.0026
25	127.70	−8 × 10⁻¹⁰

The boundary itself barely moves (x ≈ 1.09, close to LDA’s 1.098), but the coefficients, their standard errors and their p-values become meaningless, and software either warns or stops at an arbitrary iteration. Separation is common with many predictors and few observations: in the 20-predictor, 40-observation scenario of §7, 1 of 100 training sets was perfectly separable. Penalised fits (ridge or lasso; Firth’s correction, 1993) give finite estimates. LDA, which never maximises this likelihood, has no such problem.

6. Generative Classifiers: Bayes’ Theorem

Logistic regression models $Pr (Y = k ∣ X)$ directly. The other route models how each class produces its data, the density $f_{k} (x)$ of $X$ within class $k$ and the class’s share $π_{k}$ , and inverts it with Bayes’ theorem:

Pr (Y = k ∣ X = x) = \frac{π _{k} f _{k} ( x )}{\sum _{l = 1}^{K} π _{l} f _{l} ( x )} .

The predicted class is the one with the largest $π_{k} f_{k} (x)$ . If the densities and priors were known exactly, this would be the Bayes classifier of What a Model Learns, the best possible one.

LDA with one predictor

Linear discriminant analysis (Fisher, 1936) assumes Gaussian densities with a common variance $σ^{2}$ . Taking logs, the class with the largest discriminant score

δ_{k} (x) = x \frac{μ _{k}}{σ ^{2}} - \frac{μ _{k}^{2}}{2 σ ^{2}} + lo g π_{k}

wins, a linear function of $x$ . For two classes the boundary is where the scores are equal:

x^{*} = \frac{μ _{0} + μ _{1}}{2} + \frac{σ ^{2} lo g ( π _{0} / π _{1} )}{μ _{1} - μ _{0}},

the midpoint when the priors are equal, shifted away from the more common class otherwise. In practice $μ_{k}$ , $σ^{2}$ and $π_{k}$ are estimated from the training data.

A worked example. Companies that issued a dividend had a mean profit of $\overset{ˉ}{X} = 10$ % last year, the others 0%, both with variance 36, and 80% of companies issued one. For a company with $X = 4$ , with $f_{k}$ the normal density:

Pr (dividend ∣ X = 4) = \frac{0.8 e ^{- (4 - 10)^{2} /72}}{0.8 e ^{- (4 - 10)^{2} /72} + 0.2 e ^{- (4 - 0)^{2} /72}} = \frac{0.485}{0.485 + 0.160} = 0.752.

(The lab’s “dividend example” button sets up the same case.)

QDA and naive Bayes

With $p$ predictors, LDA uses multivariate Gaussians with class-specific means and one shared covariance matrix, and the boundary stays linear. Two variations relax or tighten that assumption:

Quadratic discriminant analysis (QDA) gives each class its own covariance matrix; the $lo g$ of the determinant and the quadratic term no longer cancel, and the boundary becomes quadratic.
Naive Bayes assumes the predictors are independent within each class, $f_{k} (x) = \prod_{j} f_{kj} (x_{j})$ . With Gaussian $f_{kj}$ this is a diagonal covariance per class, but the idea is more general: each $f_{kj}$ can be any one-dimensional density, a histogram for a numerical predictor or a table of frequencies for a categorical one, which is how naive Bayes mixes both kinds. The independence assumption is usually false and the probabilities it produces are often poorly calibrated, but the classifications can still be good (Domingos & Pazzani, 1997; Hand & Yu, 2001).

The flexibility shows in the number of parameters to estimate. For $K$ classes and $p$ predictors (besides the $K - 1$ priors):

Method	Means · covariance parameters	p = 20, K = 2
LDA	$K p$ · $p (p + 1) /2$ , shared	40 + 210 = 250
QDA	$K p$ · $K p (p + 1) /2$	40 + 420 = 460
Gaussian naive Bayes	$K p$ · $K p$ variances	40 + 40 = 80

7. Which Classifier When?

Three simulated scenarios, 100 training sets each, test error rates on 5,000 new points:

Scenario	Logistic · LDA	QDA · naive Bayes
Two Gaussian classes, same covariance, n = 100 (Bayes error 0.240)	0.244 · 0.244	0.247 · 0.245
Opposite correlations (+0.7 and −0.7), n = 100	0.347 · 0.345	0.239 · 0.355
20 independent predictors, n = 40 (Bayes error 0.217)	0.372 · 0.341	0.500 · 0.327

When the shared-covariance assumption holds, logistic regression and LDA find almost the same boundary and nearly reach the Bayes error; QDA pays a little for its unused flexibility.
When the classes have different covariances, the best boundary is curved, and only QDA can follow it: its error is a third lower.
With many predictors and few observations, estimation error dominates. QDA cannot work at all: 20 observations per class cannot determine a 20 × 20 covariance matrix (it is singular), and its error is chance level. Naive Bayes, with the fewest parameters, does best, although its independence assumption is exactly true here only by construction.

The general lesson is the bias–variance trade-off again: fewer assumptions mean more parameters, which pays off only with enough data. Logistic regression and LDA usually behave similarly (Ng & Jordan, 2002, compare the two routes); LDA is more efficient when its Gaussian assumption holds, logistic regression more robust when it does not, and only LDA survives perfect separation.

8. Measuring a Classifier

The confusion matrix

Every classifier with a threshold produces four counts:

The error rate alone can be badly misleading when one class is rare. On the credit data, logistic regression with balance and student status, at the usual threshold of 0.5:

Classifier (10,000 customers, 5.1% default)	Error rate · false negative rate	False positive rate · precision
”Nobody defaults”	0.051 · 1.000	0.000 · –
Logistic, threshold 0.5	0.042 · 0.690	0.007 · 0.704
Logistic, threshold 0.2	0.058 · 0.355	0.042 · 0.456
Logistic, threshold 0.1	0.097 · 0.214	0.091 · 0.319

At 0.5, the model beats the trivial classifier by less than one point of error rate and still misses 69% of the defaulters. Lowering the threshold catches more of them at the price of more false alarms and a higher overall error rate. Which threshold is right depends on what each kind of mistake costs: a missed defaulter may cost far more than a declined good customer. The threshold is a decision, not a property of the model.

The ROC curve and AUC

Plotting the true positive rate against the false positive rate for every threshold gives the ROC curve (receiver operating characteristic). The diagonal is random guessing; a perfect classifier reaches the top-left corner. The area under the curve (AUC) summarises all thresholds in one number, and it has a direct meaning (Hanley & McNeil, 1982): it is the probability that a randomly chosen positive case gets a higher score than a randomly chosen negative one. In the experiment the trapezoidal area and the share of correctly ordered (positive, negative) pairs were both 0.8540. The credit model’s AUC was 0.931.

Two common readings need care:

“Below the diagonal is worse than guessing.” It is worse than guessing only as it stands: a classifier with AUC 0.3 ranks the classes systematically the wrong way round, and flipping its predictions gives AUC 0.7. AUC 0.5 is the real “no information” value.
AUC does not see the class balance. The ROC curve depends only on the score distributions within each class, so it is unchanged when positives become rare (Fawcett, 2006). That is a strength for comparing models, and a weakness for judging one: with 1% positives, a small false positive rate can still mean most alarms are false. Precision, or the precision–recall curve, shows this (Saito & Rehmsmeier, 2015).

9. Two Worked Examples

Logistic regression. A logistic regression of getting an A on hours studied ( $X_{1}$ ) and undergraduate GPA ( $X_{2}$ ) gives $\hat{β}_{0} = - 6$ , $\hat{β}_{1} = 0.05$ , $\hat{β}_{2} = 1$ . A student who studies 40 hours with a GPA of 3.5:

\overset{p}{^} = σ (- 6 + 0.05 \times 40 + 1 \times 3.5) = σ (- 0.5) = 0.378.

For a 50% chance the log-odds must be 0: $- 6 + 0.05 h + 3.5 = 0$ , so $h = 50$ hours.

Bayes’ theorem. The dividend example of §6: 0.752.

10. Common Claims, Checked

Claim	Verdict
”The logistic function maps rational numbers to [0, 1].”	It maps all real numbers to the open interval (0, 1): it never reaches 0 or 1. σ(0) = 0.5 exactly.
”Linear regression on a 0/1 outcome is the same as LDA.”	Same direction (cosine 1.000000), different cut-off: test error 0.158 vs. 0.154 with 80/20 classes. And 28.5% of its fitted “probabilities” were negative.
”β₁ is the change in probability per unit of X.”	It is the change in log-odds; e^β₁ is the odds ratio. The effect on the probability depends on where you start: the first $1,000 of balance adds 0.9 points, the second 64.
”A predictor’s effect has one sign.”	Students: +1.26 alone, −0.83 at equal balance. Both are right; they condition on different things.
”Logistic regression is unstable when the classes are well separated.”	With perfect separation the maximum-likelihood estimate does not exist: the slope reached 128 after 25 iterations and keeps growing. Use a penalty, or LDA.
”QDA has Kp(2p + 1) parameters.”	QDA estimates K mean vectors (Kp) and K covariance matrices (K·p(p+1)/2): 460 for p = 20, K = 2, against 250 for LDA.
”Naive Bayes assumes the features are independent.”	Independent within each class (conditional independence). A diagonal covariance is the Gaussian case; naive Bayes can use any per-feature density and mix numerical and categorical features.
”QDA is more flexible, so it is better.”	Only with enough data: it won with opposite correlations (0.239 vs. 0.345) and failed completely (0.500) with 20 predictors and 20 points per class.
”A low error rate means a good classifier."	"Nobody defaults” had an error rate of 0.051; the model at threshold 0.5 had 0.042 and missed 69% of defaulters.
”The threshold is 0.5.”	It is a choice set by the costs of the two errors; at 0.1 the model missed 21% of defaulters instead of 69%.
”A ROC curve below the diagonal is worse than guessing.”	It ranks the classes backwards; flipping it gives a curve above the diagonal (AUC 1 − AUC). 0.5 is the no-information value.
”AUC is the accuracy of the classifier.”	AUC is the probability that a random positive outscores a random negative (0.8540 both ways in the experiment); it ignores the threshold and the class balance.

Summary

Idea	In one line
Why not least squares	Invented order for K ≥ 3; fitted “probabilities” outside [0, 1] for K = 2
Logistic regression	Linear log-odds, odds ratios, maximum likelihood; no estimate under perfect separation
Confounding	A coefficient’s sign can flip when another predictor enters
Generative classifiers	πₖ fₖ(x) and Bayes’ theorem; LDA (shared covariance, linear), QDA (per class, quadratic), naive Bayes (conditionally independent)
Choosing	More flexibility pays only with enough data; separation favours LDA
Measuring	Confusion matrix, threshold as a decision, ROC and AUC, precision for rare classes

Next: Cross-Validation and the Bootstrap: Error and Uncertainty from One Sample: estimating test error and standard errors by reusing the data, what CV actually estimates, and the mistake that makes noise look perfect.

The experiments: classification.py (numpy 2.1, scipy 1.18, seeded), which implements every classifier used here. The lab runs the same algorithms in the browser: on fixed test data, its logistic regression coefficients agree with the numpy version to 15 significant digits, and its LDA, QDA and naive Bayes discriminant scores to within 7 × 10⁻¹⁵, with identical classifications. The simulated panels draw their own random numbers, so their values differ slightly from the seeded runs in the text.

References

Logistic regression

Berkson, J. (1944). Application of the logistic function to bio-assay. Journal of the American Statistical Association, 39(227).
Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society B, 20(2).
McCullagh, P., & Nelder, J. A. (1989). Generalized Linear Models (2nd ed.). Chapman & Hall.
Albert, A., & Anderson, J. A. (1984). On the existence of maximum likelihood estimates in logistic regression models. Biometrika, 71(1).
Firth, D. (1993). Bias reduction of maximum likelihood estimates. Biometrika, 80(1).

Discriminant analysis and naive Bayes

Fisher, R. A. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2).
Domingos, P., & Pazzani, M. (1997). On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning, 29.
Hand, D. J., & Yu, K. (2001). Idiot’s Bayes: Not so stupid after all? International Statistical Review, 69(3).
Ng, A. Y., & Jordan, M. I. (2002). On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. NeurIPS.

Evaluation

Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1).
Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8).
Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3).

Textbooks

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.). Springer.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer.

Notes

Explorer