Statistics

Statistics is the art of working backwards from data to the unknown process that produced it. Probability tells you what data to expect from a known coin; statistics tells you what coin you've probably got after watching it flip. This page takes you from the mean of a small dataset all the way to why cross-entropy loss in a neural network is really just maximum likelihood wearing a funny hat.

1. Why statistics — the bridge from data to claims

If you have read the probability page, you saw the forward story. You start with a known model — a fair coin, a Gaussian with known mean and variance, a roll of a twenty-sided die — and you compute what data it would produce. That direction is called generative. It's the direction physics and simulations love.

Real life almost never hands you the model. Real life hands you data. You get a spreadsheet, a log file, a clinical trial, a dataset of user clicks, or a batch of token-level losses from your latest language-model training run. Something generated that data, but you don't know what. Statistics is the inverse problem:

That inversion is surprisingly hard because it's under-determined. Many different generating processes can produce the same-looking data, and many different datasets can be drawn from the same process. So statistics cannot ever hand you a single right answer. It hands you a best guess plus an honest description of how uncertain the guess is. That pairing — estimate and uncertainty — is the product. Everything on this page is a tool for producing one or the other.

You will use statistics any time you need to answer questions like these:

• "Is the new version of the model actually better, or did it just get lucky on the benchmark?"
• "Our A/B test shows a 3% lift. How sure are we that it isn't noise?"
• "We fit a line to 200 points. How confident should we be in the slope?"
• "This classifier gets 82% accuracy on the test set. What's the plausible range of its real accuracy?"
• "We saw 4 heads in 10 flips of a coin. Is the coin biased?"

Every one of those is the same question wearing different clothes: given a finite sample, what can you honestly say about the underlying population? This is the question statistics answers, and the answer is always probabilistic.

A useful habit: whenever you see a number reported (accuracy, click-through, revenue, loss), silently ask two follow-up questions. "Compared to what?" and "How sure are you?" Statistics is the machinery that makes the second question answerable. The first question is just good taste.

2. Vocabulary cheat-sheet

Here's the minimal symbol table you'll see on this page. Keep this tab open; it will save you flipping back.

A small nomenclatural warning. Anywhere you see a Greek letter ($\mu$, $\sigma$, $\theta$), it is almost always a population parameter — unknown, theoretical, something you'd like to pin down. Anywhere you see a Latin letter with a hat or a bar ($\bar{X}$, $\hat{\theta}$, $s$), it is a function of your actual data — known, computable, and used to estimate its Greek counterpart. This Greek-vs-Latin convention is not universal but it is nearly universal, and keeping it straight will save you a lot of confusion.

One more notational quirk you'll see. Capital letters like $X$ are random variables (not-yet-observed), lower-case $x$ are specific realized values (already-observed). So when we write $\mathbb{E}[\bar{X}] = \mu$, we're making a statement about the sample mean as a random variable, before we see the data. After you see the data, $\bar{X}$ becomes a specific number $\bar{x}$, and the statement stops being a probability claim. It's a subtle but useful distinction — especially when reading older texts where the convention is enforced rigorously.

3. Descriptive statistics — summarizing what you have

Before you infer anything, you describe. Descriptive statistics is the honest first pass: compress a dataset of a thousand numbers into a handful of summaries that capture "where the data sits" and "how spread out it is". No probability, no model, no claim about the world — just an accurate summary of the numbers in front of you.

3.1 Measures of center

Given a dataset $x_1, x_2, \dots, x_n$, the three classic measures of "typical value":

The median is the middle value once you sort the data. Half the points are below it, half above. For an even $n$, average the two middle values. The mode is the most common value — useful mostly for categorical data.

3.2 Measures of spread

Knowing the center isn't enough. Two datasets can share the same mean yet look nothing alike — one tightly clustered, one wildly scattered. You need a spread measure. The canonical one is variance:

3.3 Quantiles and the IQR

For skewed or heavy-tailed data, mean and variance can be misleading. Quantiles are robust alternatives. The $q$-th quantile is the value below which a fraction $q$ of the data lies. The 0.5 quantile is the median. The 0.25 and 0.75 quantiles are the first and third quartiles, $Q_1$ and $Q_3$. The interquartile range is:

4. Sample vs population — the fundamental distinction

Here is the single most important idea in statistics. Get this right and most of the rest clicks into place.

An estimator is itself a random variable — not a fixed number. Why? Because if you collected a different sample from the same population, you'd get a different value. Every time you sample, $\bar{X}$ jumps around. Across all possible samples of size $n$, $\bar{X}$ has its own distribution. We'll meet that distribution in the next section; it's the heart of almost everything.

4.1 Bias of an estimator

An estimator is unbiased if, on average across all possible samples, it hits the right answer:

The sample mean $\bar{X}$ is an unbiased estimator of $\mu$. This is not an accident — it's the reason we prefer it. The sample variance with the $n-1$ divisor is also unbiased for $\sigma^2$. The sample variance with $n$ is biased (too small by a factor $(n-1)/n$), which is exactly why people correct it.

Unbiasedness is nice but it isn't everything. An unbiased estimator can still have enormous variance — it can be correct on average but wildly off on any single sample. You also want a consistent estimator: one whose value converges to the truth as $n$ grows. Formally, $\hat{\theta}_n \to \theta$ in probability as $n \to \infty$. The sample mean is both unbiased and consistent. In general, you care most about the estimator's mean squared error:

5. Sampling distributions and the CLT

We just said $\bar{X}$ is a random variable — it takes different values across different samples. What distribution does it have?

This is called the sampling distribution of the mean. It tells you, over the space of all possible samples of size $n$, how $\bar{X}$ is distributed around the true $\mu$. Two facts about it are so useful they're worth tattooing on the inside of your eyelids.

5.1 The mean and variance of $\bar{X}$

No matter what distribution the data is drawn from, as long as $X_1, \dots, X_n$ are independent and identically distributed with mean $\mu$ and variance $\sigma^2$:

5.2 Standard error vs standard deviation

The standard deviation of the sample mean has its own name:

In practice you don't know $\sigma$. You estimate it from the sample and use $s/\sqrt{n}$. This is what NumPy, SciPy, and every statistics textbook actually compute.

5.3 The Central Limit Theorem

Here is the result that makes statistics work at all. Take any distribution with finite mean $\mu$ and finite variance $\sigma^2$ — any. Doesn't matter how weird it is. Skewed, discrete, bimodal, whatever. Draw $n$ independent samples from it. Compute the mean. Do that many times. Then:

The CLT is why "standard error $= \sigma/\sqrt{n}$" is such a loadbearing formula. It's the width of the Gaussian that your estimate is drawn from. Whenever someone hands you a confidence interval or a p-value for a mean, they are almost always using a CLT approximation under the hood.

6. Interactive figure — sampling distribution narrowing

Here's the CLT made visible. The underlying population is a deliberately non-Gaussian mixture — a little skewed, a little lumpy, so you can't mistake it for a bell curve. We draw many samples of size $n$ from it and plot the histogram of the resulting sample means. As you crank $n$ up, two things happen:

1. The distribution of $\bar{X}$ narrows — the standard error shrinks by $1/\sqrt{n}$.
2. The distribution of $\bar{X}$ becomes Gaussian — even though the population is not.

Drag the slider. The orange bar histogram is the raw population (fixed). The blue histogram is the sampling distribution of the mean for the chosen $n$. The dashed curve is the Gaussian $\mathcal{N}(\mu, \sigma^2/n)$ that the CLT predicts.

Notice how even at $n = 5$ the blue histogram already looks mostly bell-shaped, and the dashed Gaussian is a decent fit. By $n = 30$ it's indistinguishable. By $n = 200$ the bell is tiny — nearly all the mass is within a tight window around the true $\mu$. This is what "$1/\sqrt{n}$ convergence" looks like. It's slow (squared-root slow) but it's inevitable, for absolutely any reasonable underlying distribution.

7. Estimation — point and interval

So you have a sample and want to say something about a parameter. There are two basic flavors:

• Point estimate. A single number — "the slope is 1.73". Easy to report, but strips out all the uncertainty.
• Interval estimate. A range — "the slope is somewhere in [1.52, 1.94] with 95% confidence". More honest but requires more work to compute.

7.1 Building a confidence interval

The classical recipe for a 95% confidence interval for the mean, using the CLT:

For small samples, where you don't trust the Gaussian approximation, you replace 1.96 with a slightly larger number from the Student's t-distribution with $n-1$ degrees of freedom. SciPy does this for you. For $n$ bigger than about 30, the difference is small.

The t-distribution is named after William Sealy Gosset, who worked at the Guinness brewery in Dublin in the early 1900s and needed a way to reason about the quality of small samples of hops. Guinness wouldn't let him publish under his own name so he used the pseudonym "Student". A hundred years later every undergraduate statistics class teaches "Student's t-test" and almost nobody remembers why it isn't called Gosset's. The moral of the story, to the extent there is one, is that a lot of statistics was invented by practical people trying to solve real problems on small budgets.

7.2 What a confidence interval really means (and doesn't)

8. Maximum Likelihood Estimation

Confidence intervals are great when you're estimating a mean. What about more complicated parameters? The Maximum Likelihood Estimator (MLE) is the workhorse answer. It's a general recipe: given a model with parameter $\theta$, pick the $\theta$ that makes your observed data the most plausible.

8.1 The likelihood function

Suppose your data $x_1, \dots, x_n$ is drawn independently from a distribution $p(x \mid \theta)$. The likelihood is the same function with the roles of data and parameter swapped:

8.2 Log-likelihood

Products are awful to differentiate. Taking logs turns them into sums:

8.3 The MLE recipe

1. Write down the log-likelihood $\ell(\theta)$ for your model.
2. Take $\frac{d\ell}{d\theta}$.
3. Set it to zero and solve for $\theta$. That's your $\hat{\theta}_{\text{MLE}}$.
4. (Optional) Check the second derivative is negative so you found a max, not a min.

8.4 Worked example — Gaussian MLE

Suppose $x_1, \dots, x_n \sim \mathcal{N}(\mu, \sigma^2)$ and you want MLEs for $\mu$ and $\sigma^2$. The Gaussian density is:

The log-likelihood, after summing over $n$ observations, simplifies to:

9. Hypothesis testing

Estimation asks "what is $\theta$?". Testing asks "is $\theta$ equal to some specific value, or not?". This is the framework behind A/B testing, drug trials, and "is the model improvement statistically significant?" debates.

9.1 The basic setup

You pick two hypotheses about the world:

• Null hypothesis $H_0$ — the "boring" claim. Usually "the two groups are the same" or "$\theta = \theta_0$".
• Alternative hypothesis $H_1$ — the "interesting" claim. Usually "$\theta \ne \theta_0$" or "$\theta > \theta_0$".

You then compute a test statistic from your data — a number that would be small if the null were true and large if the alternative were. Then you ask: how likely is a test statistic this extreme, if $H_0$ were the actual truth?

You pre-commit to a significance level $\alpha$, usually $0.05$, and reject $H_0$ if $p < \alpha$. The $\alpha$ is the rate at which you're willing to wrongly reject a true null — more on that in a second.

9.2 t-test worked example

The most famous hypothesis test: are two sample means different? Say group A has $n_A$ observations with mean $\bar{x}_A$ and variance $s_A^2$, and similarly for group B. The two-sample t-statistic is:

9.3 Type I and Type II errors, and power

Testing is a binary decision based on uncertain data, so two kinds of error are possible:

• Type I error — you reject $H_0$ when it was actually true. False positive. Probability = $\alpha$.
• Type II error — you fail to reject $H_0$ when it was actually false. False negative. Probability = $\beta$.

The power of a test is $1 - \beta$ — the probability of correctly detecting a real effect. It depends on the true size of the effect, the sample size, the noise level, and your choice of $\alpha$.

10. The p-value debate

P-values get abused more than probably any other concept in statistics. A brief list of things a p-value is not:

Since about 2005 a loose coalition of statisticians (Ioannidis' "Why Most Published Research Findings Are False", the ASA's 2016 statement on p-values, the replication crisis in psychology and biomedicine) has been pushing for researchers to de-emphasize p-values in favor of effect sizes, confidence intervals, and pre-registration. The p-value isn't broken, but treating $p < 0.05$ as a magic passing grade is. The recommendation: use p-values as a single piece of evidence among many, never as the whole story, and always report effect sizes and uncertainty intervals alongside.

11. Linear regression

Almost every interesting statistical question — effect of a drug dose on blood pressure, relationship between study hours and grades, the slope of the Chinchilla loss curve — can be framed as "fit a line". Linear regression is the simplest and oldest version of that, and absolutely fundamental.

11.1 The model

You have paired data $(x_1, y_1), \dots, (x_n, y_n)$. You posit the model:

11.2 Least-squares loss

The fit that minimizes the sum of squared residuals:

Taking derivatives with respect to $\beta_0$ and $\beta_1$ and setting them to zero gives the normal equations:

11.3 Least squares is MLE under Gaussian noise

Here's a beautiful connection. Under the Gaussian noise model above, the log-likelihood of the data is (up to constants not depending on $\beta$):

12. Bias-variance tradeoff

Fitting a line to data is a model with two parameters. What if your data actually curves? You could fit a parabola. What if it wiggles? A 10th-degree polynomial. What if it really wiggles? A 100th-degree polynomial. Each increase in complexity reduces training error — but past some point, test error starts getting worse. This is the bias-variance tradeoff, the most important idea in machine learning after "gradient descent".

Start with the mean-squared error of an estimator $\hat{f}(x)$ of a true function $f(x)$, averaged over the noise and over the randomness in the training data. A bit of algebra gives the famous decomposition:

"Overfitting" in machine learning is this decomposition talking. A neural network with billions of parameters fit to a small dataset has near-zero bias (it can interpolate anything) but astronomical variance (it memorizes noise). Regularization, dropout, data augmentation, early stopping, weight decay — all of these are techniques that push you back down toward a higher-bias-but-lower-variance regime, trading some expressiveness for generalization.

A curiosity from the deep-learning era: the classical U-shape of test error versus model complexity (high bias on the left, high variance on the right, sweet spot in the middle) has a second descent for modern overparameterized networks. Fit a small network to MNIST and you see the textbook shape. Fit a very large one and the test error drops again past the interpolation threshold. This "double descent" phenomenon (Belkin et al., 2019) is an active research area and a reminder that bias-variance intuition is load-bearing but not the whole story. In the strongly overparameterized regime, implicit regularization from the optimizer itself does some of the work.

13. Bayesian vs frequentist — the two schools

Everything above is the frequentist school. It treats parameters as fixed-but-unknown and talks about probabilities of data given parameters. The Bayesian school goes the other way: parameters are random variables with their own distributions, and you compute probabilities of parameters given data via Bayes' rule:

Informally: frequentist methods are sharper, cheaper, and good defaults when data is abundant and you'd rather not commit to a prior. Bayesian methods shine when data is scarce, when you genuinely do have prior knowledge to inject, and when you want to marginalize uncertainty through a complex model. Modern ML is a curious mix — stochastic gradient descent is frequentist in spirit, but ensembling, dropout, and Bayesian neural networks import Bayesian ideas whenever they help. Pick the tool that gets you an honest answer, not the tribe.

One last fact worth knowing. In the limit of lots of data, Bayesian and frequentist answers converge. The posterior concentrates around the maximum-likelihood estimate, and credible intervals become numerically indistinguishable from confidence intervals. The two schools argue loudest precisely where it matters most — in the small-data regime, where your prior (or lack of it) is doing real work. "Big data" mostly makes the philosophy moot; "small data" brings it roaring back.

14. Code — NumPy and SciPy

Here's a working tour of the page's main operations. Fit a line by hand and with numpy.polyfit, build a confidence interval, run a t-test, and do a bootstrap.

import numpy as np

# Dataset: response times in ms from a web service.
rt = np.array([112, 98, 131, 104, 120, 89, 140, 115, 102, 155])

# Measures of center
mean   = np.mean(rt)
median = np.median(rt)

# Measures of spread. Use ddof=1 for the unbiased (n-1) estimator.
var_n1 = np.var(rt, ddof=1)
std_n1 = np.std(rt, ddof=1)

# Quantiles
q1, q3 = np.percentile(rt, [25, 75])
iqr    = q3 - q1

print(f"n        = {len(rt)}")
print(f"mean     = {mean:.2f} ms")
print(f"median   = {median:.2f} ms")
print(f"std (s)  = {std_n1:.2f} ms")
print(f"IQR      = {iqr:.2f} ms")

# Standard error of the mean
se = std_n1 / np.sqrt(len(rt))
print(f"SE(mean) = {se:.2f} ms")

import numpy as np

# Fake data: y = 2.3 x + 1.5 + noise
rng = np.random.default_rng(0)
x = np.linspace(0, 10, 50)
y = 2.3 * x + 1.5 + rng.normal(0, 1.2, size=x.shape)

# Method 1: normal equations by hand
x_bar, y_bar = x.mean(), y.mean()
beta1 = np.sum((x - x_bar) * (y - y_bar)) / np.sum((x - x_bar) ** 2)
beta0 = y_bar - beta1 * x_bar
print(f"hand:    slope={beta1:.3f}  intercept={beta0:.3f}")

# Method 2: numpy.polyfit (same answer, up to rounding)
b1, b0 = np.polyfit(x, y, deg=1)
print(f"polyfit: slope={b1:.3f}  intercept={b0:.3f}")

# Predict and report R^2
y_hat = beta0 + beta1 * x
ss_res = np.sum((y - y_hat) ** 2)
ss_tot = np.sum((y - y_bar) ** 2)
r2 = 1 - ss_res / ss_tot
print(f"R^2 = {r2:.3f}")

import numpy as np
from scipy import stats

rng = np.random.default_rng(42)

# Two groups with different means (treatment vs control)
control   = rng.normal(loc=100, scale=15, size=50)
treatment = rng.normal(loc=107, scale=15, size=50)

# 95% CI for the control mean, using the t-distribution.
m  = control.mean()
s  = control.std(ddof=1)
n  = len(control)
se = s / np.sqrt(n)
# Two-sided 95% t critical value with n-1 dof:
tcrit = stats.t.ppf(0.975, df=n - 1)
ci_low  = m - tcrit * se
ci_high = m + tcrit * se
print(f"control mean = {m:.2f}  95% CI = [{ci_low:.2f}, {ci_high:.2f}]")

# Welch's two-sample t-test: are the means different?
t_stat, p_val = stats.ttest_ind(treatment, control, equal_var=False)
print(f"t = {t_stat:.2f}, p = {p_val:.4f}")
if p_val < 0.05:
    print("reject H0 at alpha=0.05 — groups differ")
else:
    print("fail to reject H0 — no detected difference")

import numpy as np

# Bootstrap: empirical sampling distribution of any statistic,
# by resampling the data with replacement.
rng = np.random.default_rng(7)
data = rng.gamma(shape=2.0, scale=3.0, size=200)  # skewed!

def bootstrap_ci(data, stat_fn, B=5000, alpha=0.05):
    n = len(data)
    boots = np.empty(B)
    for b in range(B):
        idx = rng.integers(0, n, size=n)
        boots[b] = stat_fn(data[idx])
    lo, hi = np.percentile(boots, [100 * alpha / 2, 100 * (1 - alpha / 2)])
    return stat_fn(data), lo, hi

est, lo, hi = bootstrap_ci(data, np.median)
print(f"median = {est:.3f}  95% bootstrap CI = [{lo:.3f}, {hi:.3f}]")

# Bootstrap is magic: it works for any statistic, including ones
# with no closed-form sampling distribution (median, quantiles,
# correlation, ratio of two means). When you're unsure, bootstrap.

15. Why this all matters for ML

Machine learning is applied statistics with extra compute. Every standard loss function in deep learning is a disguised maximum-likelihood estimator under some noise model.

• Mean squared error loss. MLE for regression under Gaussian noise. The squared-residual form of least squares is the Gaussian log-likelihood with the additive constants and $\sigma^2$ dropped.
• Cross-entropy loss. MLE for classification under a categorical (multinomial) noise model. Minimizing $-\sum_i y_i \log \hat{p}_i$ is exactly maximizing the log-likelihood of the observed labels under the model's softmax.
• Binary cross-entropy. MLE under a Bernoulli observation model. Same family, two outcomes.
• KL divergence. Minimizing KL from the empirical distribution to a model is again MLE — the two differ only by the data-dependent entropy that doesn't affect the argmin.

"Overfitting" is also a statistics word in disguise. A neural network that perfectly fits 10,000 random-label MNIST examples has zero training error but test accuracy at chance. In the bias-variance language, you've driven bias to zero at the cost of colossal variance. Regularization, data augmentation, and early stopping are all techniques that deliberately add bias to reduce variance, exactly the tradeoff the decomposition suggested.

When you read about "held-out validation sets", "k-fold cross-validation", "bootstrap ensembling" — you are reading about techniques for estimating the variance of an estimator without being able to re-sample from the true population. You only have one dataset, so you simulate "drawing different samples" by resampling or splitting what you have. This is the frequentist imagination applied to practical ML: pretend you could rerun the experiment, estimate the answer's wobble from the rerun, and report that wobble honestly. Every "error bar" you've ever seen on a benchmark chart is an application of this idea.

One more connection worth calling out. When you hear the phrase "statistical significance" used in an ML paper — "Model A beats Model B, $p < 0.01$" — that is a t-test (or more often a bootstrap test) over random seeds or evaluation splits. Treating each seed as an independent draw and comparing the distribution of scores gives you exactly the hypothesis-testing framework from section 9. A lot of ML papers don't do this; they just report one number per model and hope. As datasets plateau and gains get smaller, that's becoming untenable, and the field is finally learning to report uncertainty.

Once you see this, the foundation models page starts to read differently: the Kaplan and Chinchilla scaling laws are statements about how training loss (a log-likelihood average) improves with sample size $D$ and parameter count $N$. The irreducible $L_\infty$ floor is the entropy of the data itself — the $\sigma^2$ of the bias-variance decomposition. Every scaling curve is a picture of "how fast does the variance of my MLE shrink relative to the bias I'm leaving on the table?"

16. Summary

• Statistics is the inverse of probability: data → model instead of model → data. You always get an estimate plus an uncertainty statement.
• Sample vs population is the central distinction. Parameters (Greek) describe the population and are usually unknown. Estimators (Latin with hat) are functions of the sample that approximate parameters.
• The sample mean is an unbiased estimator of the population mean. Its standard error is $\sigma/\sqrt{n}$, and by the Central Limit Theorem its distribution is approximately Gaussian for any underlying population.
• Confidence intervals are ranges computed by a procedure that covers the true parameter a fixed fraction of the time. They do not mean "probability the parameter is in this interval".
• Maximum likelihood is the general recipe: pick the parameter that makes your data look most plausible. Every standard ML loss function is a negative log-likelihood in disguise.
• Hypothesis tests compare data to a null hypothesis and return a p-value. Small p-values mean "data is surprising under the null"; they do not mean "the null is false with probability $p$".
• Linear regression minimizes squared residuals — which is MLE under Gaussian noise. Least squares is not a fashion choice; it's what falls out of the Gaussian assumption.
• Bias-variance decomposition says generalization error = bias² + variance + irreducible noise. Overfitting is high variance; underfitting is high bias. Regularization pushes you along this curve.
• Bayesian vs frequentist differ on whether parameters are random. Bayesians give you interpretable posteriors at the cost of a prior. Frequentists give you sharper tools at the cost of convoluted interpretations.