Orthogonality & Gram–Schmidt

The inner product, and why right angles are independence for free

Recall from Dot Product the inner product on $\mathbb{R}^n$: $u \cdot v = u_1 v_1 + \cdots + u_n v_n$. It measures length, $\|v\| = \sqrt{v \cdot v}$, and angle — in particular $u \cdot v = 0$ exactly when $u$ and $v$ are orthogonal (perpendicular). A set is orthogonal if every pair of distinct vectors in it dots to zero.

Here's the first gift, and it's a good one. An orthogonal set of nonzero vectors is automatically linearly independent. Proof — one clean line of dotting, no row reduction required. Suppose $\{v_1, \dots, v_k\}$ are nonzero and mutually orthogonal, and

$c_1 v_1 + c_2 v_2 + \cdots + c_k v_k = 0.$

Dot both sides with $v_i$. Every cross term $v_j \cdot v_i$ ($j \ne i$) vanishes by orthogonality, leaving

$c_i (v_i \cdot v_i) = 0.$

Since $v_i \ne 0$, we have $v_i \cdot v_i = \|v_i\|^2 > 0$, so $c_i = 0$. This works for every $i$, so all coefficients are zero — independent. $\blacksquare$ No row reduction, no determinant. Perpendicularity is a stronger, cleaner reason for independence than the general case ever gives you.

Orthonormal bases: coordinates for free

Normalize each vector to length one (divide by its norm) and you have an orthonormal set: mutually perpendicular and unit length, $q_i \cdot q_j = 0$ for $i \ne j$ and $q_i \cdot q_i = 1$. An orthonormal basis $\{q_1, \dots, q_n\}$ is the luxury tier, and here's why.

In a general basis, finding the coordinates of $v$ means solving a linear system $Pc = v$. In an orthonormal basis, you skip all of that. Write $v = \sum c_i q_i$ and dot with $q_j$:

$v \cdot q_j = \sum_i c_i (q_i \cdot q_j) = c_j,$

because every term dies except $i = j$, where $q_j \cdot q_j = 1$. So

$c_j = v \cdot q_j.$

The coordinate is just a dot product. No system, no inverse, no elimination. Fuck Gaussian elimination — a single multiply-and-sum is all you need. That's the whole reason orthonormal bases are worth the trouble of building — they turn coordinate-finding from a computation into a reflex.

Projection onto a subspace: the closest point

Take a subspace $W$ with orthonormal basis $\{q_1, \dots, q_m\}$. The orthogonal projection of a vector $b$ onto $W$ is

$\operatorname{proj}_W(b) = (b \cdot q_1)\,q_1 + \cdots + (b \cdot q_m)\,q_m.$

Each term $(b \cdot q_i) q_i$ is the shadow of $b$ along one axis; sum the shadows. For a single direction $a$ (not necessarily unit), the formula is the un-normalized version you've seen:

$\operatorname{proj}_a(b) = \frac{b \cdot a}{a \cdot a}\, a.$

The deep fact: the projection is the closest point in $W$ to $b$. Among all vectors in the subspace, $\operatorname{proj}_W(b)$ minimizes the distance $\|b - w\|$, because the error $b - \operatorname{proj}_W(b)$ comes out perpendicular to $W$, and a perpendicular dropped to a subspace is the shortest segment to it. Minimizing distance to a subspace is exactly the least-squares problem — the backbone of fitting lines to noisy data — and it's nothing but a projection. Drag the vectors and watch the projection slide to the nearest point on the line, the error arrow staying perpendicular:

Gram–Schmidt: subtract the shadow, repeat

Now the main event. Given any basis $\{v_1, v_2, v_3, \dots\}$ — possibly a skewed, ugly, embarrassing mess — Gram–Schmidt manufactures an orthogonal basis spanning the same space. The move is always the same: take the next vector, subtract off its shadow on everything you've already built, keep the remainder. What's left is perpendicular to all the earlier ones by construction. Simple as hell, powerful as anything in this course.

$u_1 = v_1,$ $u_2 = v_2 - \frac{v_2 \cdot u_1}{u_1 \cdot u_1}\, u_1,$ $u_3 = v_3 - \frac{v_3 \cdot u_1}{u_1 \cdot u_1}\, u_1 - \frac{v_3 \cdot u_2}{u_2 \cdot u_2}\, u_2,$

and so on. Each subtracted term is a projection — a shadow — and removing it leaves a vector orthogonal to that direction. At the end, normalize each $u_i$ to get an orthonormal basis. Worked in full in Example 3. It's tedious arithmetic, not deep, but get one sign wrong and the orthogonality silently dies — so always check the dot products are zero at the end. That check is free and it has saved more homework than coffee has. Do not skip it.

Orthogonal matrices: the rigid motions

Stack an orthonormal basis into the columns of a matrix $Q$. Then $Q^T Q = I$ — the $(i,j)$ entry of $Q^T Q$ is exactly $q_i \cdot q_j$, which is $1$ on the diagonal and $0$ off it. A square matrix with $Q^T Q = I$ is called orthogonal, and it has a superpower: it preserves lengths and angles. For any $v$,

$\|Qv\|^2 = (Qv) \cdot (Qv) = v^T Q^T Q v = v^T I v = v \cdot v = \|v\|^2.$

Lengths are untouched; since dot products are preserved too, so are angles. Orthogonal matrices are exactly the rigid motions of space — rotations and reflections, the transformations that move the world without bending, stretching, or shearing it. Their determinant is $\pm 1$ ($+1$ rotation, $-1$ reflection). And $Q^{-1} = Q^T$ for free — inverting becomes transposing.

This is the last piece of equipment before the summit. The spectral theorem says every real symmetric matrix can be diagonalized by an orthogonal $Q$ — perpendicular axes, no distortion, no shit. You just built every single tool that sentence needs. Go do the gauntlet, then climb.