Foundation Models

This objective looks trivial. It is trivial. Yet it is sufficient, at scale, to produce models that do translation, coding, math, analogical reasoning, and instruction following. Nothing in the loss function asks for any of that. It emerges from compression pressure: to predict the next token well on a sufficiently diverse corpus, you have to model the world that generated it.

3. Kaplan scaling laws (2020)

In 2020, Kaplan, McCandlish et al. at OpenAI did the first systematic study of how LLM loss depends on three knobs: model size $N$, dataset size $D$, and compute budget $C$. Their finding — a clean power law:

Kaplan's law also gave a formula for how to split a fixed compute budget between model size and dataset size. His conclusion: if compute doubles, scale up parameters more aggressively than tokens. GPT-3 (175B params, 300B tokens) was a textbook application.

4. Chinchilla (2022)

Two years later, Hoffmann et al. at DeepMind redid the experiment more carefully and got a different answer. They fit scaling curves at many $(N, D)$ pairs, held compute fixed, and found Kaplan had under-trained his biggest models:

With $\alpha \approx 0.34$, $\beta \approx 0.28$. The optimal split is roughly "tokens and parameters scale in proportion" — for a fixed compute budget $C = 6 N D$, you should have:

Since 2023 the pendulum has swung again: for inference-heavy deployments, even "over-training" well past the Chinchilla optimum is worth it, because a smaller overtrained model is cheaper to serve forever. Llama-3-8B trained on 15T tokens is the canonical 2024 example.

5. Interactive scaling calculator

Pick a compute budget (in FLOPs). The calculator applies Chinchilla's law to find the optimal $(N, D)$ split and estimates the resulting loss. Drag the slider and watch the frontier move.

6. Emergent abilities

One of the most interesting and contested observations in the scaling literature: some capabilities appear suddenly as you scale, rather than improving smoothly. Arithmetic, multi-step reasoning, instruction following — on plots of accuracy vs. scale they look flat near random, then sharply rise past some threshold.

The caveat, from Schaeffer et al. (2023): many apparent emergent abilities disappear when you use smoother metrics. A graded metric like "partial credit on digit-level accuracy" shows continuous improvement where the brittle "exact match accuracy" shows a cliff. So the jumps are partly an artifact of the evaluation, not the model.

What's genuinely not an artifact: in-context learning, chain-of-thought reasoning, and tool use all require some minimum scale before they're even discoverable. The 125M parameter GPT-2 cannot be prompted into solving arithmetic no matter how you phrase the prompt. Something qualitative happens around 7B–70B parameters that makes those capabilities accessible.

7. Adaptation — turning a base into an app

A pretrained base model is not, by itself, a useful product. It will cheerfully continue any text, including things you didn't want it to say. Adaptation turns it into an assistant. Three standard layers:

1. Supervised fine-tuning (SFT). Train on a few tens of thousands of high-quality instruction–response pairs. Cheap, effective, and mostly shapes tone and format.
2. Preference optimization. Collect pairs of outputs $(y_w, y_l)$ where $y_w$ is preferred to $y_l$, and train the model to prefer $y_w$ over $y_l$. Flavors: RLHF (PPO on a learned reward), DPO (closed-form supervised objective), IPO, KTO. Shapes values, helpfulness, safety.
3. Lightweight adapters. LoRA, QLoRA, IA³ — instead of updating all billions of parameters, train a tiny add-on module (~1% of params) for a specific task. The base stays frozen. You can serve thousands of LoRA adapters for different customers on a single shared base.

The division of labor is important: pretraining teaches the model the world. Fine-tuning teaches it the interface — how to be a chatbot, how to follow a format, what domain to privilege. It's much cheaper to do the second step; a 7B model can be fine-tuned on a single GPU in an afternoon.

8. Source code

A minimal next-token pretraining loop (no mixed precision, no distributed, no tricks), and a Chinchilla calculator.

import torch, torch.nn.functional as F

def pretrain_step(model, batch, optimizer):
    # batch: dict with input_ids (B, T+1)
    ids = batch["input_ids"]
    x = ids[:, :-1]                             # (B, T) — inputs
    y = ids[:, 1:]                              # (B, T) — targets, shifted

    logits = model(x)                              # (B, T, V)
    loss = F.cross_entropy(
        logits.reshape(-1, logits.size(-1)),
        y.reshape(-1),
    )
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    return loss.item()

# That's it. Loop this a few trillion times on the internet and a foundation
# model falls out. Every other detail is scale, data, and optimization tricks.

import math

def chinchilla_optimal(C):
    # Given compute budget C (FLOPs), return (N_opt, D_opt) per Chinchilla.
    # Rule of thumb: N ~ (C / 6)^0.5, D ~ (C / 6)^0.5, with D ≈ 20 N.
    # Under C = 6 N D, this gives N = sqrt(C / 120), D = 20 N.
    N_opt = math.sqrt(C / 120)
    D_opt = 20 * N_opt
    return N_opt, D_opt

def chinchilla_loss(N, D, Linf=1.69, A=406.4, B=410.7,
                              alpha=0.34, beta=0.28):
    # Predict loss from N and D using the Chinchilla paper's fit.
    return Linf + A / (N ** alpha) + B / (D ** beta)

# Example: 1e22 FLOPs (a mid-size pretraining run ca. 2023)
N, D = chinchilla_optimal(1e22)
print(f"N = {N/1e9:.1f}B params, D = {D/1e9:.0f}B tokens")
print(f"loss ≈ {chinchilla_loss(N, D):.3f}")

import torch, torch.nn as nn

class LoRALinear(nn.Module):
    # A drop-in replacement for nn.Linear that adds a low-rank update:
    #    y = (W + (alpha/r) * B @ A) x
    # The base W is frozen. Only A and B are trained, ~0.1% of params.
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base  = base
        self.base.weight.requires_grad_(False)
        if self.base.bias is not None:
            self.base.bias.requires_grad_(False)
        d_in, d_out = base.in_features, base.out_features
        self.A = nn.Parameter(torch.randn(r, d_in) * 0.01)
        self.B = nn.Parameter(torch.zeros(d_out, r))   # zero-init: starts as identity
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T @ self.B.T)

9. Summary

• A foundation model is a single self-supervised base, trained at massive scale, that acts as the starting point for many downstream applications.
• The pretraining loss is almost trivially simple — next-token prediction for language, masked reconstruction for images, contrastive alignment for multimodal. Everything else is scale and data.
• Kaplan (2020) found clean power-law scaling across 7 orders of magnitude of compute. Scale param-heavy.
• Chinchilla (2022) fixed the recipe: scale parameters and tokens in roughly equal proportion. Rule of thumb — 20 tokens per parameter.
• Post-2023 pendulum: over-train small models past Chinchilla-optimal if you're going to pay serving costs forever.
• Adaptation layers — SFT, DPO/RLHF, LoRA — turn a base model into a useful application. The base is the expensive part, but you only have to do it once.