Describing Data: Scales, Averages, Spread and Correlation

Every model in machine learning starts with a table of numbers, and every mistake made reading that table carries over into the model. This is the first post of a series on the foundations of statistical learning, following the order of An Introduction to Statistical Learning (James et al., 2021): describing data, then what a model learns, regression, classification, resampling, trees, unsupervised learning and neural networks.

This post covers what can be said about data before any model is fitted: which statistics a variable allows, where its centre is, how spread out it is, and how two variables move together. Some of these points are often taught loosely; where a common statement is wrong, the post says so and shows why, and every number below comes from a seeded experiment or a classic data set.

1. Population and Sample

The population is everything a question is about: all people who commute to work in a city. A sample is the part that was measured: the 15 people who answered a survey. Descriptive statistics summarise the sample; inferential statistics use the sample to say something about the population, with a stated uncertainty.

A sample is useful when it is representative: when its composition matches the population’s in the ways that matter. Two different things decide this, and they are easy to mix up:

Property	Decided by	Fixed by more data?
Representative (unbiased)	how the units were chosen	no
Precise	how many units were measured	yes

A random sample, where every unit has the same chance of being chosen, is representative on average, whatever its size; a small random sample is merely imprecise. A large sample chosen badly stays biased. The textbook case is the Literary Digest poll of the 1936 US presidential election: it collected about 2.4 million answers, predicted Alf Landon winning with 57%, and was badly wrong: Franklin Roosevelt won 61% of the vote. Its mailing lists came from telephone directories and car registrations, and those who chose to answer differed from those who did not (Squire, 1988). George Gallup predicted Roosevelt’s win from a far smaller sample, chosen more carefully.

So “15 people is not representative because n is too small” mixes up the two. 15 randomly chosen commuters are representative but imprecise; 15 colleagues from one office are unrepresentative, and would stay so at 1,500.

2. Scales of Measurement

A variable is any attribute that can take different values: distance to work, gender, satisfaction. Which statistics make sense for it depends on its scale of measurement (Stevens, 1946):

Scale	What its values allow	Examples
Nominal	equal or different	city, shirt number, product ID
Ordinal	+ an order, but no meaningful distances	school grade, “very happy … very unhappy”, automation level
Interval	+ equal, meaningful differences, but an arbitrary zero	temperature in °C or °F, calendar year
Ratio	+ a true zero, so ratios make sense	distance, mass, age, temperature in kelvin, counts

Two consequences:

On an interval scale, ratios are meaningless. 20 °C is not “twice as warm” as 10 °C: in Fahrenheit the same temperatures are 68 °F and 50 °F, a ratio of 1.36. A difference of two interval values, though, has a true zero (no difference) and is ratio-scaled: 10 K of warming is half of 20 K.
Averaging ordinal data is a convention, not a theorem. The mean of school grades assumes the steps between grades are equal, which the scale does not guarantee; recoding the grades with different but order-preserving numbers can change which class has the higher mean. The median does not have this problem. Means of ordinal data are common and often harmless with many values and roughly even steps, but the result rests on that assumption.

Scale and discreteness are separate questions. It is tempting to say “ratio-scaled means continuous”, but a count, such as the number of siblings or the number of events per frame, has a true zero (none) and meaningful ratios (4 siblings is twice 2), yet takes only whole values. Continuous or discrete describes which values are possible; the scale describes which comparisons are meaningful.

3. Where Is the Middle? Mode, Median and Mean

Measure	Definition	Needs at least
Mode	the most frequent value (there can be several)	nominal
Median	the middle value of the sorted data (the mean of the two middle values when n is even)	ordinal
Mean	$\overset{x}{ˉ} = \frac{1}{n} \sum_{i = 1}^{n} x_{i}$	interval

The mean uses every value, which makes it efficient for clean data and fragile for dirty data. The breakdown point of an estimator is the share of values that can be made arbitrarily wrong before the estimator becomes arbitrarily wrong too (Hampel, 1971). For the mean it is 0: one bad value can move it anywhere. For the median it is 50%.

Here are 15 simulated commute distances (seeded; see the script at the end). Corrupt some of them and watch the statistics:

Statistic	Clean data	One value (22 km) mistyped as 500 km
Mean	8.78 km	40.65 km
Median	8.50 km	8.50 km
Standard deviation	5.98 km	127.16 km
Interquartile range	5.90 km	5.90 km
Median absolute deviation	4.00 km	4.00 km

One typo multiplied the mean by 4.6 and the standard deviation by 21; the median and the two robust spread measures did not move. Neither is “the right average”: for income or travel time, the mean tells you the total burden (the sum is n × mean), the median tells you what a typical person experiences. Report the one that answers the question, and look at the data when they disagree.

Quantiles generalise the median: the $p$ -quantile has a share $p$ of the values below it. The quartiles $Q_{1}, Q_{2}, Q_{3}$ are the 0.25, 0.5 and 0.75 quantiles, $Q_{2}$ is the median, and a percentile is a quantile written in percent. For small samples, software differs in how it interpolates between values (Hyndman & Fan, 1996, list nine definitions); numpy’s default, used here, interpolates linearly.

4. How Spread Out? Variance, Standard Deviation and Friends

Two data sets can share their mean, median and mode and still look nothing alike, so a measure of location needs a measure of spread beside it.

Why divide by n − 1

The variance of a population of $N$ values with mean $μ$ is the mean squared deviation, $σ^{2} = \frac{1}{N} \sum (x_{i} - μ)^{2}$ . For a sample, the population mean is unknown and the sample mean $\overset{x}{ˉ}$ is used instead. But $\overset{x}{ˉ}$ is computed from the same values, so it sits closer to them than $μ$ does, and the squared deviations come out too small. Their expected value is

E [\frac{1}{n} i = 1 \sum n (x_{i} - \overset{x}{ˉ})^{2}] = \frac{n - 1}{n} σ^{2},

so the usual sample variance divides by $n - 1$ (Bessel’s correction):

s^{2} = \frac{1}{n - 1} i = 1 \sum n (x_{i} - \overset{x}{ˉ})^{2}, s = s^{2} .

Formula sheets differ here, and so does software: numpy’s var and std divide by $n$ by default (ddof=0), pandas’ by $n - 1$ (ddof=1). The same 18 values can therefore give a standard deviation of 7.60 in one place and 7.82 in another. Neither is a bug, but only $n - 1$ estimates the population variance without bias. 200,000 samples per row from a population with variance 1:

Sample size n	Average with 1/n (expected (n − 1)/n)	Average with 1/(n − 1)
2	0.503 (0.500)	1.006
5	0.799 (0.800)	0.998
10	0.899 (0.900)	0.999
30	0.965 (0.967)	0.999

(The square root of an unbiased variance is still a slightly biased standard deviation; for most purposes the difference does not matter.)

Other measures of spread

Measure	Definition	Robust?
Standard deviation $s$	$s^{2}$ , in the units of the data	no
Interquartile range	$Q_{3} - Q_{1}$ : the range of the middle half	yes (25% breakdown)
Median absolute deviation	the median of $∣ x_{i} - median ∣$	yes (50% breakdown)
Coefficient of variation	$s / \overset{x}{ˉ}$ : spread relative to the mean, without units	no, and only on a ratio scale

The coefficient of variation divides by the mean, so it only makes sense when zero means “none”: the same five temperatures (18–26 °C, spaced by 2) have a CV of 0.144 in Celsius, 0.079 in Fahrenheit and 0.011 in kelvin. Only the kelvin value means anything, because only kelvin has a true zero.

A box plot (Tukey, 1977) draws the quartiles as a box, the median as a line, and whiskers to the most extreme values that lie within 1.5 × IQR of the box; values beyond are drawn as single points. The 1.5 is a convention that flags about 0.7% of the values of a normal distribution, not a definition of an outlier: whether a flagged value is wrong, rare or the most interesting point in the data is a question for the person who knows how it was measured.

Centring and standardising

A linear transformation $y_{i} = a + b x_{i}$ moves the mean the same way, $\overset{y}{ˉ} = a + b \overset{x}{ˉ}$ , and scales the spread by $∣ b ∣$ : $s_{y} = ∣ b ∣ s_{x}$ . Two special cases are used constantly in machine learning:

Centring, $y_{i} = x_{i} - \overset{x}{ˉ}$ , gives mean 0.
Standardising, $z_{i} = (x_{i} - \overset{x}{ˉ}) / s_{x}$ , gives mean 0 and standard deviation 1, so variables measured in different units become comparable. Many methods in this series (regularised regression, k-means, PCA, neural networks) behave very differently with and without it.

5. Histograms and Densities

A bar chart shows the frequency of each category of a discrete variable. A histogram groups a continuous variable into adjacent bins, without gaps. Drawn on the density scale, each bar’s area (not height) is the share of values in its bin:

height = \frac{share of values in the bin}{bin width}, total area = 1.

This keeps histograms with different bin widths, or different numbers of observations, comparable, and as the bins become narrower (with more data) the histogram approaches the density of the variable. The bin width is a choice, and it changes the picture: too wide hides structure such as two peaks, too narrow shows noise. Kernel density plots smooth instead of binning, and have the same trade-off in their bandwidth.

6. Two Variables at Once

Two categorical variables: contingency tables

A contingency table counts how often each combination of two categorical variables occurs. Its cells give the joint frequencies, its row and column sums the marginal distributions of each variable on its own, and dividing a row by its sum gives the conditional distribution of the column variable within that row, which is usually what a question is about (“what share of students without a car has two or more siblings?”).

Association between nominal variables is measured from the χ² statistic of the table. Pearson’s contingency coefficient $C = χ^{2} / (χ^{2} + n)$ cannot reach 1 (its maximum for a $k \times k$ table is $(k - 1) / k$ ), so values from tables of different sizes are not comparable; Cramér’s V, $χ^{2} / (n (min (r, c) - 1))$ , is scaled to [0, 1] and is the more common choice today.

Two numerical variables: covariance and correlation

The covariance measures whether two variables move together:

cov (x, y) = \frac{1}{n - 1} i = 1 \sum n (x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ}) .

Its sign gives the direction, but its size depends on the units. Dividing by both standard deviations gives Pearson’s correlation coefficient,

r_{x y} = \frac{cov ( x , y )}{s _{x} s _{y}} \in [- 1, 1],

which measures the strength of a linear relationship: $\pm 1$ when all points lie exactly on a rising or falling line. (Whether the covariance uses $n$ or $n - 1$ does not matter for $r$ , as long as the standard deviations use the same.) For ordinal data, or for any relationship that is monotone but not linear, Spearman’s ρ is Pearson’s $r$ computed on the ranks of the values, with tied values sharing their average rank.

$r$ is a summary of a cloud of points, and very different clouds share it. Anscombe’s quartet (1973) is the classic warning:

Data set	Same in all four	What differs
I	mean x 9.00, var x 11.00, mean y 7.50, var y 4.12–4.13, r = 0.816, line y = 3.00 + 0.500 x	a noisy straight line (Spearman 0.818)
II	the same	a smooth curve (Spearman 0.691)
III	the same	a perfect line except one outlier (Spearman 0.991)
IV	the same	ten points at x = 8, one at x = 19 that alone creates the slope (Spearman 0.500)

Two more cases from the experiments:

$y = e^{x}$ is perfectly monotone: Spearman ρ = 1.000, Pearson r = 0.858.
$y = x^{2}$ plus noise, for $x$ from −3 to 3: r = −0.063, although a parabola explains 96.6% of the variance of $y$ . No correlation does not mean no relationship.

“0.8 is a strong correlation” is a convention, not a rule. Labels such as ”≥ 0.8 high, ≥ 0.5 moderate” are rules of thumb, and fields use very different ones: Cohen (1988) called 0.1, 0.3 and 0.5 small, medium and large effects in the behavioural sciences, while a sensor calibration with r = 0.95 may be unusably poor. What matters is what the remaining scatter means for the question.

7. Correlation and Causation

“Correlation does not imply causation” is well known. The reverse is less often stated, and it is just as true: causation does not imply correlation. A common formulation says that correlation is a precondition for causation, so that finding no correlation rules out a causal effect. It does not, and the three ways the two can part are worth knowing:

A confounder creates a correlation without causation. In 300 simulated towns, the number of fire hydrants and the number of dogs correlate at r = 0.917, because both grow with the population. Removing the population’s effect from both (regressing each on log population and correlating what is left, the partial correlation) leaves r = 0.023.
Opposite effects create causation without correlation. An advert raises liking by 0.5 points per hour seen in one group and lowers it by 0.5 in another. Within the groups the correlations are 0.692 and −0.725; over everyone it is −0.050. The advert changes everyone’s opinion, and the pooled correlation says nothing happened.
A non-linear effect is invisible to r. Too little and too much of something (a dose, a temperature, a learning rate) can both be worse than the middle: a U-shaped effect, with r near 0 (the parabola above).

A fourth case is common in practice: restricted range. If only people above a test-score cutoff are admitted, the correlation between test score and later performance among those admitted is much weaker than in the full population, even if the test works.

So a correlation is neither necessary nor sufficient for a causal effect. Establishing causation needs either an experiment, in which the cause is assigned at random so that no confounder can be behind it, or explicit assumptions about how the variables influence each other, the subject of causal inference (Pearl, 2009). The same confounding returns in regression and classification: the classification post shows a variable whose effect reverses sign when another variable is added to the model.

8. Common Claims, Checked

Claim	Verdict
”A sample of 15 is not representative because n is too small.”	Representativeness depends on how units are chosen, precision on how many. A random sample of 15 is representative but imprecise; the Literary Digest’s 2.4 million answers were unrepresentative and predicted the wrong winner.
”A ratio-scaled variable is always continuous.”	No: counts (siblings, events, defects) have a true zero and meaningful ratios but only whole values. Scale and discreteness are separate properties.
”The mean of ordinal data is meaningless.”	It assumes equal steps between categories, which the scale does not guarantee, and it can change under an order-preserving recoding. Often acceptable in practice with that caveat; the median needs no such assumption.
”Variance is the mean squared deviation from the mean.”	For a population. A sample’s variance divides by n − 1: dividing by n underestimates by the factor (n − 1)/n (0.80 at n = 5 in 200,000 simulated samples). numpy and pandas use different defaults.
”The coefficient of variation compares spread across any variables.”	Only on a ratio scale: the same temperatures give a CV of 0.144 in °C, 0.079 in °F and 0.011 in K.
”The mean is the best average.”	It has breakdown point 0: one mistyped value moved it from 8.78 to 40.65 km while the median stayed at 8.50. Which average is right depends on the question.
”Points outside the box plot’s whiskers are outliers.”	The 1.5 × IQR rule flags about 0.7% of normally distributed values. Flagged values deserve a look, not automatic removal.
”r = 0.816 tells you how the data look.”	Anscombe’s four data sets share r = 0.816, the means, the variances and the regression line, and look completely different. Plot the data.
”r ≥ 0.8 is a strong correlation.”	A rule of thumb that differs by field: 0.5 is a large effect in psychology, 0.95 can be poor for a measuring instrument.
”No correlation means no relationship.”	r measures straight lines only: y = x² + noise gave r = −0.06 while a parabola explained 97% of the variance.
”Correlation is a precondition for causation, so no correlation rules out a causal effect.”	False: opposite effects in two groups gave r = −0.05 for a real effect, and non-linear effects and restricted ranges hide correlations too. Correlation is neither necessary nor sufficient for causation.
”Correlation does not imply causation, so a correlation tells us nothing.”	It is evidence that something connects the variables: a cause, a common cause or a selection effect. The work is in finding out which.

Summary

Idea	In one line
Sample vs. population	Bias comes from how a sample is drawn, imprecision from its size
Scales	Nominal, ordinal, interval, ratio decide which statistics are meaningful; discreteness is separate
Location	Mean (efficient, fragile), median (robust, 50% breakdown), mode (any scale)
Spread	Sample variance divides by n − 1; IQR and MAD are robust; CV needs a ratio scale
Distributions	Histograms on the density scale have total area 1; bin width is a choice
Two variables	Contingency tables (Cramér’s V), covariance, Pearson for lines, Spearman for monotone relations
Causation	Correlation is neither necessary nor sufficient; confounders, cancelling effects and curves break the link both ways

Next: What a Model Learns: Prediction, Inference and the Bias–Variance Trade-off: a function learned from examples, the difference between predicting and explaining, and why fitting the training data better can predict new data worse.

The experiments: describing_data.py (numpy 2.1, scipy 1.18, seeded). The lab computes the same statistics in the browser; on the fixed data sets (the commute distances and Anscombe’s quartet) its numbers agree with numpy and scipy to every printed digit. Its simulated panels draw their own random numbers, so their values differ slightly from the seeded runs quoted in the text.

References

Data and measurement

Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684).
Squire, P. (1988). Why the 1936 Literary Digest poll failed. Public Opinion Quarterly, 52(1).
Hyndman, R. J., & Fan, Y. (1996). Sample quantiles in statistical packages. The American Statistician, 50(4).

Robust and exploratory statistics

Hampel, F. R. (1971). A general qualitative definition of robustness. Annals of Mathematical Statistics, 42(6).
Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley.
Huber, P. J., & Ronchetti, E. M. (2009). Robust Statistics (2nd ed.). Wiley.

Correlation and association

Pearson, K. (1896). Mathematical contributions to the theory of evolution. III. Regression, heredity, and panmixia. Philosophical Transactions of the Royal Society A, 187.
Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15(1).
Cramér, H. (1946). Mathematical Methods of Statistics. Princeton University Press.
Anscombe, F. J. (1973). Graphs in statistical analysis. The American Statistician, 27(1).
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum.

Causation

Yule, G. U. (1926). Why do we sometimes get nonsense-correlations between time-series? Journal of the Royal Statistical Society, 89(1).
Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society B, 13(2).
Pearl, J. (2009). Causality: Models, Reasoning, and Inference (2nd ed.). Cambridge University Press.

Textbooks

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.). Springer.
Freedman, D., Pisani, R., & Purves, R. (2007). Statistics (4th ed.). W. W. Norton.

Notes

Explorer