What Is Principal Component Analysis? How It Works

Principal component analysis, or PCA, is a way to shrink data that has many measurements per item down to a few summary numbers while keeping most of what makes the items differ.1PubMed Central. Principal component analysis: a review and recent developments A table too wide to picture becomes something you can plot on a page and compare at a glance.2Wikipedia. Principal component analysis The method is over a century old, first described by Karl Pearson in 1901, yet it still sits underneath tools in genetics, finance, and face recognition.2Wikipedia. Principal component analysis

The Core Idea Behind PCA

Each new axis PCA finds is called a principal component, a direction drawn through the cloud of data points.2Wikipedia. Principal component analysis The first one runs along the direction in which the points spread out most, which statisticians call the largest variance.2Wikipedia. Principal component analysis Each next component takes the direction of greatest remaining spread that sits at a right angle to all the earlier ones.2Wikipedia. Principal component analysis The result is a set of uncorrelated axes, so each one captures a pattern the others don’t repeat.1PubMed Central. Principal component analysis: a review and recent developments

Imagine people plotted by height and weight. The points form a tilted oval, like a rugby ball lying at an angle. The long axis of that oval is the first component, a kind of “overall size.” The short axis across it is the second component, roughly “build for one’s height.” Drop the short axis and you keep most of the differences between people with one number instead of two.

Wanting the most spread and wanting the least loss turn out to be the same goal. The first component is also the line that sits closest, on average, to all the points.2Wikipedia. Principal component analysis That’s because the spread kept by the projection plus the information lost always adds up to the data’s total spread.3Ruhr-Universität Bochum. Lecture Notes on Principal Component Analysis So keeping as much spread as possible is exactly the same as losing as little as possible.3Ruhr-Universität Bochum. Lecture Notes on Principal Component Analysis

A component isn’t one of your original measurements. It’s a weighted mix of all of them, and a high weight means that measurement strongly shapes the component. Because components are new variables, naming one (“overall size”) takes judgment from the analyst. PCA doesn’t pick the best original columns and delete the rest. What gets dropped are the later, low-spread components, not original columns.

The new axes can also look tilted. The original axes usually line up with the columns of a table, such as height and weight, but PCA is free to rotate to any angle that fits the cloud, so its axes often run diagonally across the old ones. Rotating doesn’t stretch, squash, or change the data, and the distances between points stay the same. Only the viewpoint changes, to one where the spread is easiest to summarize.

Strongly related measurements are where PCA shines. When two measurements rise and fall together, like arm length and leg length, they carry largely overlapping information, and PCA folds that overlap into a single component instead of counting it twice. The more closely the measurements are related, the fewer components you need to keep most of the spread. If every measurement were unrelated to the others, PCA would have little to compress.

How PCA Works Step by Step

The number paired with each component is called its eigenvalue, and it measures how much spread lies along that direction.5Sebastian Raschka, PhD. Principal Component Analysis Divide one component’s eigenvalue by the total of all of them and you get the share of the data’s spread that component explains.2Wikipedia. Principal component analysis Software usually computes all this with a related shortcut called SVD, which also depends on centering the data first.4arXiv. PCA, SVD, and Centering of Data

Centering comes before anything else because, without it, the first direction tends to point at the data’s average instead of its spread. Shifting the cloud means the directions describe how items differ from the typical one. Most software centers automatically, but it is worth checking the setting.

The covariance table is a compact summary of the cloud’s shape. Each entry says whether two measurements tend to rise together, move in opposite directions, or have no consistent link, and a large positive entry means they climb together. The diagonal entries compare each measurement with itself, which gives its own spread.

Take five people measured on height and weight. Subtract the average height and the average weight from each, so the cloud sits around zero. Because tall people here also tend to be heavy, the table shows a strong positive link, and the first component points up and to the right along the oval. Each person is then described by how far along that direction they sit, one number replacing two.

That number is called a score. Each item gets a score on every kept component, and scores replace the original measurements for plotting or later analysis. Two items with similar scores are similar on the main patterns the kept components capture.

You rarely do this by hand. Spreadsheet add-ons, R, Python, and statistics packages run PCA in one command, and the output usually lists components, eigenvalues, shares of spread, weights, and scores.

How Many Components to Keep

A scree plot is a chart of eigenvalues from largest to smallest, and many people keep the components before the curve flattens into an “elbow.”6MetricGate. Reading a Scree Plot: How Many Components? Components after that point add small, similar amounts and often reflect noise. If there’s no clear elbow, lean on the other rules.

Another rule keeps only components with an eigenvalue above 1 on data put on the same scale.7Ecology. Stopping Rules in Principal Components Analysis: A Comparison of Heuristical and Statistical Approaches On such data, each original measurement contributes a spread of 1, so a component above 1 explains more than a single original measurement would. It’s quick, but with many measurements it can keep too many components.

Some analysts instead keep however many components it takes to reach a fixed share of total spread.7Ecology. Stopping Rules in Principal Components Analysis: A Comparison of Heuristical and Statistical Approaches Targets like “most of the spread” are a convention, not a law of nature, and the right one depends on how much detail the task needs. If the spread is thinly scattered over many components, the target may demand a lot of them, which tells you the data has little shared structure to compress.

A more careful check compares each eigenvalue against what purely random data of the same size would produce, and keeps only those that beat it.6MetricGate. Reading a Scree Plot: How Many Components? Even pure noise produces eigenvalues that fall from large to small, because chance alone makes some directions look wider than others.

These approaches don’t always agree on the same number.7Ecology. Stopping Rules in Principal Components Analysis: A Comparison of Heuristical and Statistical Approaches It’s sensible to try two or three and see whether they point to a similar answer.

Your purpose matters too. For a 2D chart you keep two components regardless, and you should note the share they explain. For feeding later analysis, keeping more components protects detail.

How to Read PCA Results and Biplots

A biplot shows each item as a point using its scores on the first two components.8Forrest Young’s Notes. What Is Principal Component Analysis? How It Works It also draws each original measurement as an arrow showing how strongly it shapes those components.8Forrest Young’s Notes. What Is Principal Component Analysis? How It Works Items that sit close together are similar on the measurements the main components capture.9Nature Reviews Methods Primers. Principal component analysis

Each axis is a component, and its label usually gives the share of spread that component explains, which tells you how much of the data the picture shows. The axes are new variables, so a position on the page means a mix of measurements. The arrows translate that back.

A long arrow means the measurement matters a lot to the components shown. Arrows pointing the same way mean those measurements tend to rise together, arrows at right angles are largely unrelated, and opposite arrows move in opposite directions.

Clusters of points suggest groups of similar items, and points far from everything else may be unusual items worth checking. A point far out along an arrow scores high on that measurement. To read a score, imagine dropping a line from the point onto the arrow’s direction: the further along it lands, the higher the item is on that measurement. A point on the opposite side of the center from the arrow’s tip is below average on it.

The chart shows only the first two components. If they explain a small share of spread, apparent closeness can mislead, so always check the share printed on the axes.

Preparing Your Data and Avoiding Common Pitfalls

Because PCA chases spread, a measurement recorded in large units can dominate just because its numbers are bigger.10scikit-learn. Importance of Feature Scaling In one wine dataset, a single chemical measured on a scale about 100 times larger than the others took over the first component until the data was rescaled.10scikit-learn. Importance of Feature Scaling The fix is to standardize, meaning rescale each measurement to an average of 0 and a spread of 1, when measurements use different units.11GeeksforGeeks. Principal Component Analysis (PCA)

If a result is mostly about one column, ask whether that column is truly important or just had large numbers. Checking the weights on the first component quickly reveals this.

Scaling isn’t always right. If all measurements share the same units and their size differences are meaningful, putting them on the same scale may hide real information. Pixel brightness values in images are a typical example. Decide based on whether a big number means “more important” or just “different units.”

Standard PCA is very sensitive to outliers and can give misleading results when they are present.12Taylor & Francis Online / Applied Economics Letters. Robust vs. classical principal component analysis in the presence of outliers A robust version, designed to resist extreme values, is a better choice when data contains them.12Taylor & Francis Online / Applied Economics Letters. Robust vs. classical principal component analysis in the presence of outliers First check whether an extreme value is a typing or measurement error. If so, correct or remove it, and note that you did. If it’s a real, unusual item, don’t delete it just because it’s awkward. Run PCA with and without it, and if the components change a lot, use a robust version or report both.

Standard PCA also needs a complete table, so gaps must be filled first.13Journal of Machine Learning Research. Practical Approaches to Principal Component Analysis in the Presence of Missing Values One simple method fills gaps with averages, runs PCA, uses its result to improve the guesses, and repeats until they settle.13Journal of Machine Learning Research. Practical Approaches to Principal Component Analysis in the Presence of Missing Values Filling gaps this way gave results closely matching the full data in nutrition studies, especially with smaller samples.14PubMed. Missing data imputation via the expectation-maximization algorithm can improve principal component analysis aimed at deriving biomarker profiles and dietary patterns

PCA combines measurements by simple weighted adding, so it only sees straight-line patterns. Curved or looping structure can be squashed or missed. And big spread isn’t always the interesting signal; a small component can matter.

Where PCA Is Used in Real Life

PCA is widely used in population genetics to reveal how genetic variation is structured by geography and ancestry.15PubMed Central. A Genealogical Interpretation of Principal Components Analysis Each person’s genome has thousands of markers, and PCA reduces these to a couple of scores per person. Plotted, those scores often form clusters that line up with where people’s ancestors lived, and nearby populations sit near each other, so the plot can look like a rough map. It describes shared ancestry in a sample, not a verdict on any individual.

In finance, interest rates for 30 different loan lengths don’t move independently but follow a few shared patterns.16mbrenndoerfer.com. Principal Component Analysis: Factor Extraction for Finance Those patterns are usually read as the curve’s overall level, its slope, and its bend.16mbrenndoerfer.com. Principal Component Analysis: Factor Extraction for Finance The first component moves all rates up or down together, the level. The second makes short and long rates move in opposite directions, tilting the curve. The third raises or lowers the middle relative to the ends. The first three components explain over 99% of yield curve movement,16mbrenndoerfer.com. Principal Component Analysis: Factor Extraction for Finance so a day’s move can be described with three numbers instead of thirty.

PCA has been one of the most successful techniques in face recognition, where its components are called eigenfaces.17International Journal of Artificial Intelligence & Applications. Optimizing Face Recognition Using PCA Each eigenface is itself a ghostly face-shaped image. Any face can be rebuilt as a mix of a few dozen of them, so comparing faces means comparing a short list of mixing amounts. The method can be slow on very large image databases.17International Journal of Artificial Intelligence & Applications. Optimizing Face Recognition Using PCA

PCA Versus Other Methods and Variations

PCA ignores any labels and just finds the widest spread, while LDA looks for directions that best separate groups you already know.18sesen.ai. LDA vs PCA: Supervised vs Unsupervised Reduction With no labels, PCA is the natural choice because LDA cannot run without them. With labels, LDA is aimed at your goal, since the widest spread need not be the direction that separates the groups. PCA might line up along a difference that has nothing to do with the groups you care about.

t-SNE and UMAP capture curved structure and keep nearby points together, while PCA is straight-line and preserves overall spread.19MetricGate. t-SNE vs UMAP vs PCA: Dimension Reduction PCA gives the same answer every run, while t-SNE and UMAP involve randomness.19MetricGate. t-SNE vs UMAP vs PCA: Dimension Reduction Use PCA first for a quick, repeatable overview, and switch to t-SNE or UMAP when clusters look tangled in a PCA plot. Distances between far-apart clusters in t-SNE plots aren’t reliable, unlike PCA.

Many analysts also run PCA first to cut noise and size, then feed the top components into clustering or t-SNE. That speeds up analysis and often cleans the result.

Sparse PCA forces each component to use only a few measurements, making it easier to interpret.20Journal of Computational and Graphical Statistics. Sparse Principal Component Analysis The trade is a little less spread captured for clearer meaning.

Kernel PCA first maps data into a richer space so curved patterns become straight ones it can find.21Sebastian Raschka, PhD. Kernel tricks and nonlinear dimensionality reduction via RBF kernel PCA It asks you to pick a kernel, the rule for that mapping, and different choices give different results. Because the components live in the mapped space, they can’t be written as simple weights on the original measurements, so it’s harder to explain which measurements matter.