Multivariable Calculus

A function of several variables

$f$

The function itself. You plug in numbers, you get a number out.

$x_1, x_2, \ldots, x_n$

The $n$ inputs, usually called the independent variables. Each one is a real number.

$n$

The number of inputs — the dimension of the input space. $n=2$ for a 3D surface you can draw, $n=10^9$ for a modern language model.

Why this matters. When you train a neural network, you're minimizing a scalar loss $\mathcal{L}(\theta)$ where $\theta = (\theta_1, \ldots, \theta_n)$ is the weight vector and $n$ can be billions. Every tool on this page is about how a scalar changes when you wiggle a very long vector of inputs.

Scalar vs. vector fields

$\mathbb{R}^n$

The set of all $n$-tuples of real numbers. $\mathbb{R}^2$ is the plane, $\mathbb{R}^3$ is 3D space, $\mathbb{R}^n$ for $n$ large is "parameter space."

$\to$

"Maps to." Read $f : \mathbb{R}^n \to \mathbb{R}$ as "$f$ is a rule that turns any vector of $n$ numbers into one number."

$f$

Lowercase for scalar-valued functions.

$\mathbf{F}$

Bold capital for vector-valued functions. The bold is a flag: "this returns a vector, watch yourself."

$m$

The dimension of the output. For a vector field on a surface, $m = 2$; for loss, $m = 1$.

Rule of thumb. If the thing you're asking about is a single number (loss, probability, energy, height), it's a scalar field — you can draw level curves and think about it as a landscape. If it's a direction or vector (velocity, force, neural activation vector), it's a vector field and you have to keep track of components.

Partial derivative

$\dfrac{\partial f}{\partial x_i}$

"Partial derivative of $f$ with respect to $x_i$." The curly $\partial$ says "I'm only changing one variable." Sometimes written $f_{x_i}$ or $\partial_{x_i} f$.

$\mathbf{a}$

The point at which you're evaluating. Bold because it's a vector $\mathbf{a} = (a_1, \ldots, a_n)$.

$h$

A tiny scalar — the size of the nudge you apply to coordinate $i$ only. Taking $h \to 0$ gives the instantaneous rate.

$a_i + h$

Coordinate $i$ after the nudge. Every other coordinate stays at its original value.

$\lim_{h \to 0}$

Limit as $h$ shrinks to zero — same concept as single-variable calculus, just applied in one coordinate direction.

Picture. Stand on a hillside and face due east. Walk a tiny step in that direction. How fast did your altitude change per unit of eastward distance? That number is $\partial f / \partial x$. Turn 90° and do the same thing: $\partial f / \partial y$. You haven't moved in any other direction, so every other coordinate is frozen.

Partial with respect to $x$

$f(x, y) = 3x^2 y + y^3$

The function we're differentiating. Two inputs, one output.

$6xy$

Result of treating $y$ as a constant: $\tfrac{d}{dx}(3x^2 y) = 6xy$, and $\tfrac{d}{dx}(y^3) = 0$ because $y^3$ is a constant when $y$ is frozen.

Partial with respect to $y$

$3x^2$

From $\tfrac{d}{dy}(3 x^2 y) = 3 x^2$ (because $3 x^2$ is a constant factor when $x$ is frozen).

$3y^2$

From $\tfrac{d}{dy}(y^3) = 3 y^2$, using the power rule.

Trick. Taking partial derivatives is single-variable calculus. You just temporarily pretend the other letters are constants. Every rule you know — product, quotient, chain — still works.

Clairaut's theorem / equality of mixed partials

$\dfrac{\partial^2 f}{\partial x \, \partial y}$

The mixed partial: differentiate once in $y$, then once in $x$ (right-to-left, following the conventional order).

$=$

These are equal if both mixed partials exist and are continuous near the point. Which they always are for anything you're going to meet on this site.

Why it matters. This is why the Hessian matrix (§8) is symmetric — and symmetry is what lets you diagonalize it and talk about eigenvalues, which is how you classify saddle points vs minima.

The gradient

$\nabla$

"Nabla" or "del." It's an operator (a thing that eats a function and hands back another function), and $\nabla f$ is read "grad $f$" or "del $f$."

$\nabla f(\mathbf{x})$

The gradient vector of $f$ evaluated at the point $\mathbf{x}$. It lives in $\mathbb{R}^n$ — same dimension as the input.

$\dfrac{\partial f}{\partial x_i}$

The $i$-th partial, defined in §3. One scalar per coordinate direction.

$\mathbf{x}$

The point in $\mathbb{R}^n$ where you're evaluating the gradient. Different point, different gradient.

Intuition. The gradient is the all-the-partials-at-once vector. It answers "if I take a unit step in the best direction, how much does $f$ go up, and which way is best?" — in one object.

First-order Taylor approximation

$\mathbf{v}$

A small displacement vector in $\mathbb{R}^n$. Think "tiny arrow from $\mathbf{x}$."

$f(\mathbf{x} + \mathbf{v})$

The value of $f$ at the nearby point.

$\nabla f(\mathbf{x}) \cdot \mathbf{v}$

The dot product of the gradient with the step. This is a single number — the first-order change in $f$.

$\approx$

"Approximately equal." The approximation gets better as $\|\mathbf{v}\|$ gets smaller, and the error shrinks faster than linearly.

In plain English. Near $\mathbf{x}$, the function looks like a flat tilted plane, and the tilt is given by the gradient. Any small displacement changes $f$ by (roughly) gradient-dot-displacement.

Dot product in terms of angle

$\|\nabla f\|$

The length (Euclidean norm) of the gradient vector — a non-negative scalar.

$\|\mathbf{v}\| = 1$

We're looking at unit-length steps, so this drops out.

$\theta$

The angle between $\nabla f$ and $\mathbf{v}$. This is the only thing you can still vary.

$\cos\theta$

Takes its maximum value of $1$ when $\theta = 0$ — i.e., when $\mathbf{v}$ points the same way as $\nabla f$.

Conclusion. The direction that maximizes the first-order change in $f$ is the gradient's own direction, and the maximum change is $\|\nabla f\|$. That's why "the gradient points uphill" is not a mnemonic — it's a theorem about dot products.

Directional derivative

$D_{\mathbf{u}} f$

The directional derivative of $f$ in direction $\mathbf{u}$. A single scalar — the rate at which $f$ changes per unit distance moved in direction $\mathbf{u}$.

$\mathbf{u}$

A unit vector, $\|\mathbf{u}\| = 1$. If you forget to normalize, you'll get a number that depends on the length of $\mathbf{u}$, which is meaningless. Always normalize.

$\cdot$

Dot product — see the linear algebra page.

Check. If $\mathbf{u}$ is the unit vector along the $x_i$-axis (all zeros except a $1$ in slot $i$), the dot product picks off the $i$-th component of the gradient — so $D_{\mathbf{u}} f = \partial f / \partial x_i$. Directional derivatives generalize partials; they don't replace them.

Computing the directional derivative

$(8, 3)$

The gradient of $f$ at $(1, 2)$. Two numbers, one per input.

$\tfrac{1}{\sqrt 2}(1, 1)$

The normalized "northeast" direction. Dividing by $\sqrt 2$ makes the length 1.

$11 / \sqrt 2$

The instantaneous rate of change of $f$ as you walk northeast from $(1, 2)$.

Sanity check. The rate is smaller than $\|\nabla f\| = \sqrt{64 + 9} \approx 8.54$, as it should be: moving in any direction other than the gradient's own is strictly less efficient for climbing.

Chain rule — one parameter

$z = f(x, y)$

A scalar function of two intermediate variables.

$x = x(t), \; y = y(t)$

Both $x$ and $y$ depend on the same parameter $t$.

$dz/dt$

The total rate of change of $z$ as $t$ changes. We want one number.

$\partial f / \partial x, \; \partial f / \partial y$

How fast $f$ changes when you wiggle its inputs directly. These live at the current $(x, y)$.

$dx/dt, \; dy/dt$

How fast each intermediate variable changes as $t$ changes. Single-variable derivatives.

Why the sum? Wiggling $t$ wiggles both $x$ and $y$, and each wiggle pushes $z$ through its own partial. You add the two contributions because, to first order, their effects are independent. This is the same reason gradients add components — linearity.

Chain rule — the general form

$z$

A scalar output (a loss, a cost, a probability).

$g_1, \ldots, g_n$

Intermediate variables. Each one might depend on any or all of the $t_j$. Think of them as the outputs of a hidden layer.

$t_1, \ldots, t_m$

The primitive inputs you actually control. In ML these are the network's weights and biases.

$\dfrac{\partial z}{\partial g_i}$

"Upstream gradient" — how much $z$ responds to a wiggle in the intermediate $g_i$ directly.

$\dfrac{\partial g_i}{\partial t_j}$

"Local gradient" — how much $g_i$ responds to a wiggle in the primitive input $t_j$.

$\sum_{i=1}^{n}$

Sum over all intermediate variables. If $t_j$ affects multiple $g_i$'s, each path contributes a term, and you add.

The core slogan. "Downstream sensitivity $=$ sum over paths of (upstream sensitivity $\times$ local sensitivity)." That is the entire chain rule, in any dimension, stated in words.

The Jacobian matrix

$\mathbf{f} : \mathbb{R}^n \to \mathbb{R}^m$

A vector-valued function: $n$ inputs in, $m$ outputs out.

$f_1, f_2, \ldots, f_m$

The component functions. Each $f_i$ is a scalar function of all $n$ inputs.

$\mathbf{J}_{\mathbf{f}}$

The Jacobian matrix of $\mathbf{f}$. Sometimes written $D\mathbf{f}$ or $\partial \mathbf{f} / \partial \mathbf{x}$.

Shape

$m$ rows, $n$ columns. Row $i$ is the gradient of $f_i$ (transposed); column $j$ tells you how all $m$ outputs respond to wiggling input $j$.

$\partial f_i / \partial x_j$

The partial of the $i$-th output with respect to the $j$-th input — a scalar.

Why a matrix? Because the best linear approximation to a vector function near $\mathbf{x}$ is $\mathbf{f}(\mathbf{x} + \mathbf{v}) \approx \mathbf{f}(\mathbf{x}) + \mathbf{J}_{\mathbf{f}}(\mathbf{x}) \, \mathbf{v}$, and a linear map from $\mathbb{R}^n$ to $\mathbb{R}^m$ is, by definition, an $m \times n$ matrix. The Jacobian is "the derivative as a matrix."

A concrete Jacobian

Row 1

Partials of $f_1 = x^2 y$. $\partial f_1 / \partial x = 2xy$, $\partial f_1 / \partial y = x^2$.

Row 2

Partials of $f_2 = x + \sin y$. $\partial f_2 / \partial x = 1$, $\partial f_2 / \partial y = \cos y$.

Application. Multiplying a small input change $(dx, dy)$ by this matrix gives the linear approximation to the resulting output change $(df_1, df_2)$. For the same reason the gradient is the local best-linear-fit to a scalar field, the Jacobian is the local best-linear-fit to a vector field.

Chain rule in Jacobian form

$\mathbf{f} \circ \mathbf{g}$

Function composition — "do $\mathbf{g}$ first, then $\mathbf{f}$."

$\mathbf{J}_{\mathbf{f}} \mathbf{J}_{\mathbf{g}}$

Matrix product of the two Jacobians, evaluated at compatible points.

Moral. The multivariable chain rule is just matrix multiplication. This is why backprop can be written as a sequence of matrix-vector products — each layer contributes its local Jacobian, and the product of Jacobians (left-to-right from the output side) is the gradient of the loss with respect to the inputs.

The Hessian matrix

$\mathbf{H}_f$

The Hessian of $f$. An $n \times n$ matrix of second partial derivatives.

$\partial^2 f / \partial x_i^2$

Diagonal entries — "pure" second partials. Curvature along each coordinate axis.

$\partial^2 f / \partial x_i \partial x_j$

Off-diagonal entries — "mixed" second partials. How the slope in direction $i$ changes as you move in direction $j$.

Symmetry

By Clairaut's theorem, $\partial^2 f / \partial x_i \partial x_j = \partial^2 f / \partial x_j \partial x_i$, so the Hessian is symmetric for any well-behaved $f$. This is a huge deal.

One-line slogan. The Hessian is the Jacobian of the gradient. Gradient gives you the slope as a vector function of position; take its Jacobian and you get the Hessian. That's why it's symmetric (equality of mixed partials) and that's why it matters (it's the best quadratic fit to $f$ near a point).

Second-order Taylor expansion

$\mathbf{v}$

A small displacement from $\mathbf{x}$.

$\nabla f \cdot \mathbf{v}$

First-order (linear) correction — from §4.

$\tfrac{1}{2} \mathbf{v}^\top \mathbf{H}_f \mathbf{v}$

Second-order (quadratic) correction. This is a quadratic form: a number built by sandwiching the matrix $\mathbf{H}_f$ between $\mathbf{v}$ and its transpose.

$\mathbf{v}^\top$

The transpose of $\mathbf{v}$ — turning a column vector into a row vector so the matrix product is a scalar.

$\tfrac{1}{2}$

Comes from the second-order term of a Taylor series, same as $f''(a) h^2 / 2$ in 1D.

Geometric reading. Near $\mathbf{x}$, the function looks like a tilted quadratic bowl. The tilt comes from the gradient; the shape of the bowl — whether it's a well, a dome, or a saddle — comes from the Hessian. Which is exactly what classifies critical points in §10.

2D second-derivative test

$f_{xx}, f_{yy}, f_{xy}$

The three second partials — short-hand for $\partial^2 f / \partial x^2$, $\partial^2 f / \partial y^2$, $\partial^2 f / \partial x \partial y$.

$H$

The determinant of the $2 \times 2$ Hessian. Positive $H$ means the eigenvalues have the same sign; negative $H$ means opposite signs (saddle).

$f_{xx} > 0$

Curves up in the $x$-direction. Combined with $H > 0$, that forces both eigenvalues to be positive.

Why it works. For a $2 \times 2$ symmetric matrix, $\det = \lambda_1 \lambda_2$ and $\text{tr} = \lambda_1 + \lambda_2 = f_{xx} + f_{yy}$. Same-sign eigenvalues $\Leftrightarrow \det > 0$; positive eigenvalues $\Leftrightarrow \det > 0$ and $\text{tr} > 0$, and the trace sign matches $f_{xx}$'s sign when $\det > 0$.

Gradient of the cubic

$3x^2 - 3y$

Partial w.r.t. $x$: differentiate $x^3 \to 3x^2$, $-3xy \to -3y$, $y^3 \to 0$.

$-3x + 3y^2$

Partial w.r.t. $y$: $x^3 \to 0$, $-3xy \to -3x$, $y^3 \to 3y^2$.

Hessian of the cubic

$6x$

Second partial $\partial^2 f / \partial x^2$ — differentiate $3x^2 - 3y$ again in $x$.

$-3$

Mixed partial — $\partial/\partial y$ of $3x^2 - 3y$, or equivalently $\partial / \partial x$ of $-3x + 3y^2$.

$6y$

Second partial $\partial^2 f / \partial y^2$.

Double integral as iterated integral

$R$

A region in the plane, perhaps described as "$a \le x \le b$ and $c(x) \le y \le d(x)$."

$dA$

An infinitesimal area element — the 2D analogue of $dx$. In $(x,y)$ coordinates, $dA = dx \, dy$.

$\iint_R$

"Double integral over $R$." The two integral signs mean we integrate in two dimensions.

$\int_{c(x)}^{d(x)} \, dy$

Inner integral: integrate out $y$ first, treating $x$ as fixed. Result is a function of $x$.

$\int_a^b \, dx$

Outer integral: integrate the result over $x$. Turns the function of $x$ into a number.

Fubini's theorem. For any reasonable integrand and region, you can do the two integrals in either order and get the same answer. "Reasonable" covers essentially everything on this site.

Change of variables in 2D

$\mathbf{g}$

The coordinate transformation: $\mathbf{g}(u,v) = (x(u,v), y(u,v))$.

$R'$

The pre-image of $R$ under $\mathbf{g}$ — the same region, described in $(u,v)$ coordinates.

$\mathbf{J}_{\mathbf{g}}$

The $2 \times 2$ Jacobian matrix of $\mathbf{g}$. Its determinant measures how much $\mathbf{g}$ stretches or squishes area near the point.

$|\det \mathbf{J}_{\mathbf{g}}|$

Absolute value of the determinant. The absolute value is there because orientation doesn't matter for measuring area.

Example. Polar coordinates: $x = r\cos\theta$, $y = r\sin\theta$, and $\det \mathbf{J} = r$. That's where the $r \, dr \, d\theta$ you memorized comes from — it is literally the Jacobian determinant of the polar transformation.

Lagrange condition

$f$

The objective function you want to optimize.

$g$

The constraint function. The constraint is "$g = 0$" — a level curve of $g$.

$\mathbf{x}^*$

A candidate optimum on the constraint surface.

$\lambda$

The Lagrange multiplier — an unknown scalar you solve for alongside $\mathbf{x}^*$. Its sign tells you which way along the constraint the objective is increasing.

$\nabla f = \lambda \nabla g$

"The gradient of $f$ is parallel to the gradient of $g$." Geometrically: the level curve of $f$ is tangent to the constraint curve at the optimum.

Why it works. If the level curve of $f$ crossed the constraint transversally at $\mathbf{x}^*$, you could walk along the constraint in one direction and increase $f$. So at an optimum, the two curves must be tangent, which is the same as saying their normals (i.e., their gradients) are parallel. That parallelism is exactly $\nabla f = \lambda \nabla g$.

Gradients of objective and constraint

$(y, x)$

Gradient of $xy$: partial w.r.t. $x$ is $y$, partial w.r.t. $y$ is $x$.

$(1, 1)$

Gradient of the linear constraint $x + y - 1$: both partials are 1.

A generic supervised loss

$\mathcal{L}(\theta)$

The scalar training loss as a function of the parameter vector $\theta$. This is the scalar field we're descending.

$\theta$

The parameter vector — all weights and biases stacked into one long vector in $\mathbb{R}^n$ with $n$ potentially in the billions.

$N$

The number of training examples in a batch.

$(\mathbf{x}_i, \mathbf{y}_i)$

The $i$-th training input and its target. $\mathbf{x}_i$ might be an image, $\mathbf{y}_i$ its label.

$\text{net}_\theta$

The network — a composition of many layers, each parameterised by a slice of $\theta$.

$\ell$

A per-example loss (cross-entropy, squared error, etc.). Scalar-valued.

So what do we do with it? Compute $\nabla_\theta \mathcal{L}$ (a vector the size of $\theta$), take a step in the direction $-\nabla_\theta \mathcal{L}$ (steepest descent, §4), repeat. That's the whole algorithm — gradient descent. Every improvement (momentum, Adam, second-order methods, natural gradient) is a modification to this loop that uses more information about the loss landscape to take better steps.