How Principal Component Analysis (PCA) Works Mathematically

You've got a spreadsheet with 50 columns about every customer: age, income, clicks, time on site, forty-six more. How do you find the two or three numbers that actually explain most of what's going on?

That's the problem PCA solves. And here's the surprising part: the entire method comes down to rotating a line until a shadow gets as wide as possible.

Let's build it from that one idea.

Think of your data as fireflies in the dark

Picture your data points as fireflies hovering in a dark room, each one glowing at its own (x, y) position. Now hold up a flat wall and shine a light so every firefly casts a shadow onto it.

Rotate the wall slowly. At some angles, the shadows bunch up near the middle, cramped and uninformative. At other angles, they spread out across the whole wall, some far left, some far right, everything in between.

The angle where the shadows spread out the most is the one that keeps the most information about where the fireflies actually are. That angle is the first principal component. Everything else in PCA is just making this idea precise.

Drag the line around. Notice the shadow variance climbing and falling as you rotate, and settling at one clear maximum. That maximum isn't an accident, it's the direction the data is actually stretched along.

Turning "spread" into a number

We need a way to measure how spread out those shadows are. The natural choice is variance, the average squared distance from the mean.

If $w$ is a unit vector pointing along our wall's direction, then each point $\mathbf{x}_i$ casts a shadow at position

That's just the dot product, the length of $\mathbf{x}_i$'s shadow along $w$. We write this as a projection.

So the question "which angle spreads the shadows the most?" becomes a clean optimization problem: find the unit vector $w$ that maximizes the variance of $z_1, z_2, \dots, z_n$.

No calculus needed yet, just notice that this variance depends entirely on the angle of $w$, and somewhere in that circle of angles, there's a winner.

The covariance matrix is the shape of the cloud

Here's the shortcut that makes this solvable without checking every angle by hand. All the information about how your data spreads and correlates lives in one small object: the covariance matrix.

The diagonal tells you how spread out each variable is on its own. The off-diagonal tells you how much they move together. And here's the part that connects straight back to our fireflies: if you draw the covariance matrix as an ellipse, its long axis points exactly toward the direction of maximum shadow variance.

Slide the correlation up and down. Watch the ellipse tilt and stretch, and watch its two axes stay perpendicular no matter what. Those two axes are the eigenvectors of $\Sigma$, and their lengths are the eigenvalues. The long axis is PC1. The short one, always at a right angle to it, is PC2.

Where the eigenvalue equation comes from

Now for the notation. We want to maximize $w^\top \Sigma w$ (that expression is exactly the variance of the projections) subject to $w$ staying a unit vector, $\|w\| = 1$.

Solve that constrained maximization with a bit of calculus, called a Lagrange multiplier, and something clean falls out:

That's it. That's the eigenvalue equation. It says: the best direction $w$ is one where multiplying by $\Sigma$ doesn't rotate $w$ at all, it only stretches it, by a factor of $\lambda$. And that stretch factor $\lambda$ turns out to equal the variance captured along that direction.

So "the direction of maximum shadow spread" and "the eigenvector of the covariance matrix" are the same thing. That's the whole trick behind PCA. Everything else is bookkeeping: sort the eigenvectors by their eigenvalues, biggest first, and you've ranked every possible direction from "most informative" to "least informative."

Throwing away dimensions on purpose

Here's where PCA earns its keep. If PC1 captures, say, 90% of the variance, you can describe every point using just its position along PC1, and only lose 10% of the information.

That's dimensionality reduction: project every point onto the top few components, and drop the rest.

Hit play and watch every 2D point collapse onto the single PC1 line. Notice the reconstruction error, the average squared gap between each original point and its compressed version, start at zero and grow as the compression increases. That gap is exactly the variance you're throwing away, the variance that belonged to PC2.

How many components should you actually keep?

With two dimensions, keeping one component is an easy call. But real datasets have dozens or hundreds of columns. How do you decide where to cut?

You look at how much variance each component explains, sorted largest to smallest, and add them up until you're happy. This chart is called a scree plot.

Click through the bars, or drag the slider. Notice how the first two or three components already capture most of the total variance, while the rest barely move the cumulative line. That's the pattern PCA is built to exploit: real-world data is rarely spread evenly across every dimension, it's usually stretched hard along just a few.

A common rule of thumb: keep enough components to explain 90-95% of the variance, and throw the rest away.

The short version

PCA rotates your coordinate system to point along the directions your data is actually stretched in. The best direction maximizes the variance of the data projected onto it, which is the same as finding the top eigenvector of the covariance matrix. Rank all the directions by their eigenvalues, keep the first few, and you've replaced a pile of correlated columns with a handful of numbers that capture almost everything that matters.

Fifty confusing columns become three meaningful ones. Not because you deleted information carelessly, but because you rotated your view until most of it lined up along just a few axes.

All visualizations are interactive React components running entirely in your browser. The covariance ellipse and eigenvectors use a closed-form 2x2 eigendecomposition. No libraries beyond React.