Linear Regression, Read Correctly: Least Squares, Intervals and p-Values

What a Model Learns separated prediction from inference. Linear regression is the method where the two meet most clearly: it predicts, and its coefficients come with standard errors, confidence intervals and p-values that say how much to trust them. Those numbers are also some of the most often misread in all of statistics.

This post fits linear regressions to a real data set, the fuel economy of 32 cars from 1974 (mtcars; Henderson & Velleman, 1981), and checks every interpretation against simulations in which the true coefficients are known: what “95% confidence” actually guarantees, what a p-value is and is not, why R² can mislead, and what happens to p-values when predictors are chosen by looking at them.

---

1. The Model

Simple linear regression assumes that the outcome is a straight-line function of one predictor, plus noise:

$$Y={\beta }_0+{\beta }_{1}X+\epsilon .$$

${\beta }_0$ is the intercept (the mean of $Y$ when $X=0$) and ${\beta }_1$ the slope (the change in the mean of $Y$ per unit of $X$). Both are unknown constants, estimated from data as ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$; a prediction is $\hat{y}={\hat{\beta }}_0+{\hat{\beta }}_{1}x$.

The running example: miles per gallon (mpg) against weight (in 1,000 lb) for the 32 cars of mtcars. Heavier cars use more fuel, and the question is how much more, and how sure we can be.

---

2. Least Squares

For a candidate line, the residual of observation $i$ is $e_i=y_i-{\hat{y}}_i$, its vertical distance from the line. Least squares (Legendre, 1805; Gauss, 1809) chooses the line that minimises the residual sum of squares,

$$\text{RSS}=\sum _{i=1}^n{e}_i^2=\sum _{i=1}^n(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i{)}^2,$$

which has a closed-form minimum:

$${\hat{\beta }}_1=\frac{{\sum }_i(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_i(x_i-\bar{x}{)}^2},{\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}.$$

For mtcars this gives ${\hat{\beta }}_0=37.29$ and ${\hat{\beta }}_1=-5.34$: 5.34 fewer miles per gallon for every extra 1,000 lb, on average. Try to beat it:

The RSS surface is a bowl with a single lowest point. It is a long, tilted valley, because the data sit far from weight 0: a higher intercept with a steeper slope fits almost as well, so the two estimates are strongly correlated. Centring the predictor removes this.

With several predictors, the same idea has a geometric form. Stack the outcomes into a vector $y$ and the predictors, with a column of ones, into a matrix $X$. Every choice of coefficients gives a prediction vector $X\beta$ in the space spanned by the columns of $X$, and least squares picks the one closest to $y$:

Two useful facts follow from the picture: the residuals are orthogonal to every predictor ($X^{\top }e=0$, the normal equations $X^{\top }X\hat{\beta }=X^{\top }y$), and with an intercept they sum to zero, so the fitted line passes through $(\bar{x},\bar{y})$.

---

3. What the Formulas Assume

Least squares can be computed for any data. The standard errors, intervals and p-values below, though, rest on assumptions about the noise:

Assumption	What breaks if it fails	How to check
Linearity: the mean of $Y$ is linear in the predictors	the coefficients describe the wrong shape	residuals against fitted values show a pattern
Independent errors	standard errors are too small (e.g. repeated measurements, time series)	how the data were collected
Constant variance of $\epsilon$	standard errors are wrong where the spread differs	the residuals fan out
Normal errors	only matters for exact small-sample intervals and tests	a normal quantile plot of the residuals

Linearity is the one to check first. For mtcars, adding ${\text{wt}}^2$ improves the fit (its coefficient has $p=0.003$, and R² rises from 0.753 to 0.819): the drop in mpg flattens for heavy cars. A straight line is a useful summary, not the truth. Everything below takes the model as given, which is exactly the situation in which its intervals and p-values are valid.

---

4. How Precise Is the Slope? Standard Errors and Confidence Intervals

A different sample of 32 cars would give a different ${\hat{\beta }}_1$. The standard error measures how much it would vary:

$$\mathrm{SE}({\hat{\beta }}_1{)}^2=\frac{{\sigma }^2}{{\sum }_i(x_i-\bar{x}{)}^2},{\hat{\sigma }}^2=\frac{\text{RSS}}{n-2}.$$

The slope is estimated more precisely with less noise, more data and a wider spread of $x$. For mtcars, $\mathrm{SE}({\hat{\beta }}_1)=0.559$, and the 95% confidence interval is

$${\hat{\beta }}_1\pm t_{0.975,n-2}\cdot \mathrm{SE}({\hat{\beta }}_1)=-5.34\pm 2.042\times 0.559=[-6.49,-4.20].$$

What does “95%” mean? It is tempting to say “the true slope lies in [−6.49, −4.20] with probability 95%“. In the classical (frequentist) framework this is not what the interval guarantees: the true slope is a fixed number, and this particular interval either contains it or does not. The 95% describes the procedure: computed on many samples, 95% of the intervals it produces contain the true value (Neyman, 1937). Below, 100 intervals at a time, from data with a true slope of 0.5:

(A statement of the form “95% probability that the slope lies in this range” is a Bayesian credible interval, which needs a prior distribution for the slope; with flat priors it often coincides numerically with the confidence interval, but it answers a different question.)

”± 2 standard errors” is an approximation. The exact multiplier is the 97.5% quantile of a t distribution with $n-2$ degrees of freedom, which is close to 2 only for moderate $n$. Measured over 20,000 simulated data sets per row:

n	Exact t multiplier · coverage	Coverage with ± 2 SE · ± 1.96 SE
5	3.182 · 95.0%	85.8% · 85.1%
10	2.306 · 94.9%	91.8% · 91.3%
30	2.048 · 95.0%	94.4% · 93.8%
100	1.984 · 94.9%	95.1% · 94.6%

With 5 observations, a “95%” interval built with ± 2 SE misses the truth one time in seven.

---

5. Is There a Relationship? t-Tests and p-Values

To test the null hypothesis $H_0:{\beta }_1=0$ (no linear relationship) against $H_1:{\beta }_1\ne 0$, compute how many standard errors the estimate is from 0:

$$t=\frac{{\hat{\beta }}_1-0}{\mathrm{SE}({\hat{\beta }}_1)}.$$

If $H_0$ is true (and the assumptions hold), $t$ follows a t distribution with $n-2$ degrees of freedom. The p-value is the probability, computed under $H_0$, of a $t$ at least as far from 0 as the one observed. For mtcars, $t=-9.56$ and $p=1.3\times 10^{-10}$: such a strong slope would almost never arise by chance if weight were unrelated to mpg.

Five things a p-value is not (Wasserstein & Lazar, 2016; Greenland et al., 2016):

1. It is not the probability that $H_0$ is true. It is computed assuming $H_0$ is true. Turning it into $\mathrm{Pr}(H_0\mid \text{data})$ needs a prior.
2. It is not the size or importance of an effect. With 100,000 observations, a true slope of 0.02 (estimated 0.024), explaining 0.06% of the variance (R² = 0.00058), had $p=2.6\times 10^{-14}$.
3. A large p-value is not evidence of no effect, only a lack of evidence for one; small samples often cannot detect real effects.
4. “Significant at 5%” is a convention. When $H_0$ is true, p-values are uniform between 0 and 1: in 20,000 data sets with no effect, 4.97% had $p<0.05$, and 0.97% had $p<0.01$, exactly as designed.
5. The thresholds for $t$ depend on the degrees of freedom. The often-quoted ”$|t|>2$ for 5%, $|t|>2.75$ for 1%” holds for about 30 degrees of freedom:

Degrees of freedom	$|t|$ for p = 0.05	$|t|$ for p = 0.01
10	2.228	3.169
30	2.042	2.750
∞ (normal)	1.960	2.576

In simple regression, the t-test of the slope and the test of the correlation coefficient are the same test: for mtcars both give $p=1.29\times 10^{-10}$, and $R^2=r^2=0.7528$.

---

6. How Well Does the Line Fit? RSE and R²

Two summaries describe the fit as a whole. The residual standard error $\text{RSE}=\sqrt{\text{RSS}/(n-p-1)}$ estimates $\sigma$, the typical size of a residual, in the units of $Y$: 3.05 mpg for mtcars. The coefficient of determination

$$R^2=1-\frac{\text{RSS}}{\text{TSS}},\text{TSS}=\sum _i(y_i-\bar{y}{)}^2,$$

is the share of the variance of $Y$ explained by the model: 0.753 for weight alone. Three properties are often forgotten:

• R² never falls when a predictor is added, even a column of random noise: the least-squares fit can always ignore it, and usually uses it a little.
• Adjusted R², $1-\frac{\text{RSS}/(n-p-1)}{\text{TSS}/(n-1)}$, charges for every parameter and can fall.
• R² lies between 0 and 1 only on the training data of a model with an intercept. On new data, a model that predicts worse than the mean of $Y$ has negative R².

One real predictor plus $k$ columns of pure noise, 50 observations, averaged over 2,000 data sets:

Noise predictors k	Training R² · adjusted R²	Test R² on new data
0	0.276 · 0.261	0.232
10	0.425 · 0.259	0.029
20	0.578 · 0.262	−0.342
40	0.881 · 0.269	−4.144

---

7. Multiple Regression

With $p$ predictors the model is $Y={\beta }_0+{\beta }_1{X}_1+\cdots +{\beta }_p{X}_p+\epsilon$, still fitted by least squares (now with matrix algebra). Adding horsepower to the mtcars model:

Coefficient (mpg ~ wt + hp)	Estimate · SE	t · p-value
Intercept	37.23 · 1.60	23.28 · 2.6 × 10⁻²⁰
Weight (1,000 lb)	−3.878 · 0.633	−6.13 · 1.1 × 10⁻⁶
Horsepower	−0.0318 · 0.0090	−3.52 · 0.0015

(RSE 2.59, R² = 0.827, adjusted 0.815; F = 69.2 with $p=9\times 10^{-12}$.)

Interpreting a coefficient. ${\hat{\beta }}_j$ is the average difference in $Y$ between observations that differ by one unit in $X_j$ and have the same values of the other predictors. The weight coefficient changed from −5.34 alone to −3.88 with horsepower in the model, because heavier cars also tend to be more powerful ($r=0.66$): part of what weight “explained” alone was horsepower. This is the confounding of Describing Data, handled by putting the confounder in the model.

Two cautions that the phrase “holding the others fixed” hides:

• It is an association, not an effect. Nothing in observational data says what would happen if one car were made heavier. Causal claims need an experiment, or explicit causal assumptions.
• “Fixed” may not exist. When predictors are strongly correlated, data with one varying and the others fixed are rare, and the coefficient is estimated from little information.

The F-test: is any predictor useful?

$$F=\frac{(\text{TSS}-\text{RSS})/p}{\text{RSS}/(n-p-1)}$$

tests $H_0:{\beta }_1=\cdots ={\beta }_p=0$ in a single test. Individual t-tests cannot replace it: with many predictors, some will have small p-values by chance.

Collinearity

Adding displacement and the number of cylinders to the model, both strongly related to weight and horsepower, gives an F-test with $p=10^{-10}$, while three of the four predictors are individually “not significant” (displacement $p=0.33$, horsepower 0.10, cylinders 0.06). The predictors carry overlapping information, so the data cannot tell them apart, and their standard errors grow. The variance inflation factor ${\text{VIF}}_j=1/(1-R_j^2)$, where $R_j^2$ comes from regressing $X_j$ on the other predictors, measures this: 10.4 for displacement. Simulated, with two predictors of correlation ρ:

Correlation ρ	Mean SE of β̂₁	√VIF
0	0.102	1.00
0.9	0.234	2.29
0.99	0.722	7.09

---

8. Correlation and Regression

In simple regression, a significant slope and a significant correlation are the same thing (identical p-values, above). In multiple regression they come apart. In this simulation $y=x_1-0.7x_2+\text{noise}$, with $x_1$ and $x_2$ correlated at 0.7, so that $x_2$‘s direct effect and its path through $x_1$ cancel in the marginal correlation:

Model	Coefficient of $x_2$	p-value
correlation of $y$ and $x_2$	r = 0.006
y ~ x₂	0.005	0.93
y ~ x₁ + x₂	−0.680 (true −0.7)	3.9 × 10⁻³¹

$x_2$ is uncorrelated with $y$ and one of its strongest predictors: a suppressor variable (Conger, 1974). Screening predictors by their correlation with the outcome, before a multiple regression, would have thrown it away.

The two also answer different questions. The correlation is a unit-free strength of a linear relationship; the slope says how much $Y$ changes per unit of $X$ and depends on the units of both. Standardising the variables makes slopes comparable across predictors.

---

9. Choosing Predictors, and Why p-Values Stop Working

With $p$ candidate predictors there are $2^p$ possible models. Common shortcuts are forward selection (start with none and add the most useful predictor while it helps), backward selection (start with all and remove the least useful; impossible when $p>n$) and mixtures of both. A widespread version decides “useful” by p-values: add a predictor while its p-value is below 0.05.

That makes the final p-values meaningless. At each step, the procedure picks the smallest of many p-values and reports it as if it were a single test. With 20 predictors of pure noise and 100 observations (2,000 data sets):

Forward selection, “enter if p < 0.05”	Result
Data sets with at least one “significant” predictor	64.8% (expected 1 − 0.95²⁰ = 64.2%)
Mean number of predictors selected	1.06
Median reported p-value of the first one chosen	0.020

Freedman (1983) made the same point with screening regressions. The fixes are to judge models by an estimate of their test error (a validation set or cross-validation, the resampling post; or AIC and BIC), to use methods designed for many predictors (the lasso), or, when inference after selection is the goal, methods built for it (Berk et al., 2013).

---

10. Categorical Predictors

A categorical predictor with $K$ levels enters the model as $K-1$ dummy variables (0/1 indicators); the level without a dummy is the baseline, represented by the intercept, and each dummy’s coefficient is the difference from it. With an intercept, one dummy per level would make the columns linearly dependent (the “dummy variable trap”). For mtcars, cylinders (4, 6 or 8) as two dummies, with 4 cylinders as the baseline:

mpg ~ wt + cylinders	Estimate · p-value	Meaning
6 cylinders	−4.26 · 0.005	vs. a 4-cylinder car of the same weight
8 cylinders	−6.07 · 0.001	vs. a 4-cylinder car of the same weight
cylinders as a number	−1.51 · 0.001	assumes the steps 4 → 6 → 8 are equal

Coding the categories as the numbers 4, 6, 8 forces equal effects per step (here −3.0 per two cylinders), while the dummies find a larger first step (−4.26) than second (−1.82). For nominal categories, numeric codes are simply meaningless.

---

11. Predicting a New Value

A prediction has three sources of error: the estimated coefficients (reducible, shrinks with more data), the model’s form if it is wrong (bias), and the noise $\epsilon$ (irreducible). Two intervals capture the first and the third:

A 3,000 lb car (mpg ~ wt)	95% interval	Uncertainty about
Predicted mean mpg: 21.25	[20.12, 22.38]	the average of all 3,000 lb cars (confidence interval)
One particular car	[14.93, 27.57]	a single car, including its own noise (prediction interval)

The prediction interval is more than five times wider, and it does not shrink to zero with more data: it always includes $\sigma$.

---

12. Common Claims, Checked

Claim	Verdict
”The true value lies in the 95% confidence interval with probability 95%.”	In the frequentist sense, the 95% belongs to the procedure: 95% of intervals computed this way contain the truth (95.0% in 20,000 simulations). One computed interval either does or does not. The probability statement is a Bayesian credible interval, which needs a prior.
”A 95% interval is β̂ ± 2 SE.”	An approximation: the exact multiplier is a t quantile (3.18 at n = 5). With ± 2 SE, coverage was 85.8% at n = 5 and 91.8% at n = 10.
”	t
”A small p-value means the null hypothesis is probably false.”	A p-value is computed assuming the null is true; it is not the probability that it is.
”A small p-value means a strong effect.”	A slope explaining 0.06% of the variance had p = 2.6 × 10⁻¹⁴ with 100,000 observations.
”A large t means the null hypothesis is wrong.”	It means the data would be unusual if it were true. With 20,000 null data sets, 4.97% gave p < 0.05.
”R² is between 0 and 1, and higher is better.”	Only on training data with an intercept. Training R² rose from 0.28 to 0.88 with 40 noise columns while test R² fell to −4.1. Use adjusted R² or test error.
”βⱼ is the effect of a one-unit increase in Xⱼ, holding the others fixed.”	It is an association among observations with equal other predictors, not a causal effect, and is poorly determined when predictors are collinear (VIF 10.4 for displacement in mtcars).
”A significant regression requires a significant correlation.”	True in simple regression (identical tests). In multiple regression, a predictor with r = 0.006 had t = −13.9 (a suppressor).
”Forward selection with p < 0.05 finds the relevant predictors.”	On 20 noise predictors it reported at least one “significant” predictor in 64.8% of data sets. p-values after selection are not valid p-values.
”Numbers are fine for a categorical predictor.”	Only if the steps are truly equal. Dummies for 6 and 8 cylinders found unequal steps (−4.26, then −1.82).
”The prediction interval shrinks with more data.”	The confidence interval for the mean does; the interval for a single new value always includes the noise σ: [14.9, 27.6] vs. [20.1, 22.4] for a 3,000 lb car.

---

Summary

Idea	In one line
Least squares	Minimise the RSS; the fit is a projection, residuals ⟂ predictors
Assumptions	Linearity, independence, constant variance (normality for small-sample exactness); check residuals
Standard errors and intervals	95% is a property of the procedure; use the t quantile, not ± 2
p-values	P(data this extreme | H₀), not P(H₀), not effect size
Fit	RSE in units of Y; R² only rises; adjusted and test R² do not
Multiple regression	Coefficients are conditional associations; F-test; collinearity inflates SEs
Selection	p-values after selection are not p-values; use test error
Categories and prediction	K − 1 dummies; prediction intervals include the noise

Next: Classification: Logistic Regression, Discriminant Analysis and the ROC Curve: predicting a category, two routes to the class probability, a coefficient that flips sign, and why a low error rate can describe a useless classifier.

The experiments: linear_regression.py (numpy 2.1, scipy 1.18, seeded), with the mtcars data in mtcars.py; the mtcars results match the published values (mpg ~ wt: 37.285 − 5.345 wt, R² = 0.7528; mpg ~ wt + hp: R² = 0.8268). The lab computes its regressions in the browser; its two-sided t-test p-values agree with scipy to a relative error of 1.4 × 10⁻¹², and its standard error for horsepower in mpg ~ wt + hp reproduces 0.0090 (p = 0.00145). The simulated panels draw their own random numbers, so their values differ slightly from the seeded runs in the text.

---

References

Least squares and regression

• Legendre, A.-M. (1805). Nouvelles méthodes pour la détermination des orbites des comètes. Firmin Didot.
• Gauss, C. F. (1809). Theoria motus corporum coelestium in sectionibus conicis solem ambientium. Perthes & Besser.
• Galton, F. (1886). Regression towards mediocrity in hereditary stature. Journal of the Anthropological Institute, 15.
• Henderson, H. V., & Velleman, P. F. (1981). Building multiple regression models interactively. Biometrics, 37(2).

Inference

• Student [Gosset, W. S.] (1908). The probable error of a mean. Biometrika, 6(1).
• Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd.
• Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society A, 236(767).
• Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2).
• Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4).

Predictors, collinearity and selection

• Conger, A. J. (1974). A revised definition for suppressor variables: A guide to their identification and interpretation. Educational and Psychological Measurement, 34(1).
• Belsley, D. A., Kuh, E., & Welsch, R. E. (1980). Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. Wiley.
• Freedman, D. A. (1983). A note on screening regression equations. The American Statistician, 37(2).
• Berk, R., Brown, L., Buja, A., Zhang, K., & Zhao, L. (2013). Valid post-selection inference. Annals of Statistics, 41(2).
• Harrell, F. E. (2015). Regression Modeling Strategies (2nd ed.). Springer.

Textbooks

• James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.). Springer.