What Are Word Embeddings and How Do They Work?

A word embedding is a way of representing words as lists of numbers so that words with similar meanings end up near each other in a shared mathematical space. Instead of treating every word as an isolated label, embeddings place words on a kind of map where proximity reflects similarity. The idea traces back to a simple linguistic insight: words that appear in similar contexts tend to mean similar things. That principle, combined with modern computing power, has reshaped how machines handle language and now extends well beyond text into biology, social networks, and even brain science.

The Core Idea Behind Word Embeddings

Before embeddings, the standard way to represent words for a computer was to give each word its own unique slot in a very long list, with a 1 in that slot and 0s everywhere else. If your vocabulary had 50,000 words, each word was a string of 50,000 numbers with only a single 1. This approach treats “dog” and “puppy” as no more related to each other than “dog” and “refrigerator.” There is no built-in notion of meaning.

Word embeddings solve this by compressing each word down to a much shorter list of numbers, typically a few hundred. These numbers are not hand-crafted; they are learned by an algorithm that reads enormous amounts of text and notices which words tend to show up in each other’s company. The result is a dense numerical fingerprint for every word, and words that get used in similar ways end up with similar fingerprints. A review of embedding techniques describes this progression from sparse one-hot encoding to dense representations like Word2Vec, GloVe, and fastText as one of the foundational shifts in modern language processing.¹

How the Major Algorithms Work

Three algorithms dominate the first generation of word embeddings, and each takes a slightly different approach to learning those numerical representations from raw text.

Word2Vec, released by researchers at Google in 2013, comes in two flavors. The continuous bag-of-words model tries to predict a target word from the words surrounding it, while the skip-gram model does the reverse: given a single word, it tries to predict the surrounding context. Both models learn word vectors as a byproduct of getting better at these prediction tasks. To make training practical on large vocabularies, Word2Vec introduced optimization tricks like negative sampling, which speeds things up by only updating a small random subset of word vectors at each step rather than the entire vocabulary.²

GloVe, developed at Stanford a year later, took a different route. Instead of scanning text one window at a time, GloVe first counts how often every pair of words co-occurs across the entire text collection, then factors that giant table of counts into smaller vectors. The creators framed the distinction neatly: methods for learning word vectors broadly split into global matrix factorization approaches and local context-window approaches, and GloVe was designed to combine the strengths of both.³

FastText, from Facebook’s AI lab, added a twist by breaking words into smaller pieces. Rather than treating “unhappiness” as a single token, fastText looks at chunks like “un,” “happi,” and “ness” and builds the word’s vector from its parts. This makes it far better at handling words it has never seen before, because it can still assemble a reasonable guess from familiar subword pieces. Research has confirmed that these subword-level models have a clear advantage on tasks related to word structure and on datasets with high rates of unfamiliar vocabulary.⁴

The Famous Analogy Trick

The result that made word embeddings famous is the analogy test. Take the vector for “king,” subtract “man,” add “woman,” and the closest vector you land near is “queen.” This was not something the algorithms were explicitly taught; it emerged from the patterns in the training text. The vectors seem to encode abstract relationships as directions in space, so the “direction” from man to woman is roughly the same as the direction from king to queen.

Researchers have studied this phenomenon extensively and found that models like Word2Vec and GloVe construct embeddings based on how often words co-occur, and the resulting vectors not only group semantically similar words but also exhibit this striking linear analogy structure.⁵ The theoretical explanation for why this works is still an active area of research. The practical upshot, though, is that these vectors carry meaning in a way that earlier word representations never could.

When One Meaning Is Not Enough

The first-generation models have a significant blind spot: every word gets exactly one vector, no matter how many meanings it has. The word “bank” gets the same representation whether you mean a river bank or a financial institution. These are called static embeddings because the vector is fixed once training is done.

Contextual embeddings, introduced starting around 2018, changed this. Models like ELMo, BERT, and GPT generate a different vector for a word each time it appears, depending on the sentence around it. ELMo pioneered the approach by learning deep contextualized word representations that captured both syntax and semantics, modeling how word usage varies across different linguistic contexts.⁶ BERT and GPT pushed the idea further, and the shift from static to contextual embeddings has produced large performance improvements across a wide range of language tasks.⁷

How contextual are these representations in practice? Research comparing BERT, ELMo, and GPT-2 found that the degree of context-dependence varies considerably across layers and models. Words in higher layers of these networks shift more dramatically based on their surroundings than words in lower layers, meaning the models gradually transform a word from a generic representation into a highly situation-specific one.⁸

Bias Baked Into the Vectors

Because embeddings learn from human-written text, they absorb human prejudices. If the training data consistently associates certain professions with one gender or certain adjectives with one ethnic group, those associations become part of the geometry. The word for “nurse” might sit closer to “woman” and “doctor” closer to “man,” not because that reflects reality, but because the text the model read skewed that way. Research has documented that online texts across genres and styles are riddled with stereotypes, and word embeddings trained on these texts perpetuate and amplify those biases, propagating them to any machine learning system that uses the embeddings as input.⁹

One striking use of this property is as a historical measurement tool. Researchers trained embeddings on text from each decade of the 20th and 21st centuries and found that the shifting distances between words tracked real demographic and cultural changes over time, including the women’s movement in the 1960s and waves of Asian immigration into the United States.¹⁰ The embeddings essentially became a mirror of evolving societal attitudes, captured in the geometry of word relationships.

Debiasing these vectors is an active and somewhat contentious subfield. Early methods tried to project out the “gender direction” from the embedding space, but this proved harder than it sounds. Frequency patterns in the training data interfere with simple geometric fixes. A technique called Double-Hard Debias addresses this by first cleaning out frequency-related noise before removing the gender subspace, which preserves useful meaning while reducing gender bias more effectively than earlier approaches.¹¹ Other work has focused on preserving legitimate gender information (the word “queen” should remain associated with femininity) while stripping away stereotypical associations, distinguishing between four types of gender-related information to surgically target only the discriminative biases.¹² Compared to the amount of research on gender bias in embeddings, the study of racial bias remains relatively underdeveloped, with researchers still working out how to reliably construct racial dimensions in the embedding space.¹³

Crossing Language Barriers

If words in English and words in Spanish are each embedded in their own spaces, a natural question is whether you can line those spaces up so that translation becomes a matter of finding the nearest neighbor. This is exactly what cross-lingual word embeddings do. Recent work has shown that accurate alignments between monolingual embedding spaces can be obtained with little or even no supervision, sometimes needing just a small bilingual dictionary as a starting point and sometimes needing nothing at all.¹⁴

Extending this idea beyond two languages to many simultaneously is trickier. One approach learns a shared space for multiple languages at once, aligning all of them into a common coordinate system rather than chaining pairwise mappings together.¹⁵ Doing the same thing with contextual embeddings is harder still, because the vectors are not fixed and cannot be straightforwardly rotated to match up. One method works around this by first building context-independent summaries of each language’s embedding space and using those as an anchor to align the context-dependent representations.¹⁶

The Hubness Problem in High Dimensions

Embeddings live in high-dimensional spaces, and high-dimensional spaces have strange geometric properties that can quietly cause problems. One of the best-documented is hubness: some vectors end up being the nearest neighbor of many other vectors, while most vectors are the nearest neighbor of almost nothing. Think of it as a few popular kids in a massive school who keep showing up as everyone’s closest match, crowding out less prominent entries.

Research on sentence-level embeddings from BERT-based models found that hubness distorts similarity searches, making some texts appear relevant to queries they have little to do with.¹⁷ In multilingual settings, hubness has been identified as a bigger driver of retrieval asymmetry than anisotropy (the tendency for vectors to cluster in a narrow cone). Replacing standard cosine similarity with hubness-aware retrieval metrics has been recommended as a practical fix for multilingual embedding pipelines.¹⁸

Embeddings Beyond Words

The embedding idea has proven so flexible that researchers now apply it to things that look nothing like sentences. In biology, DNA and protein sequences can be treated as strings of characters, and the same techniques used to embed words can embed those biological “words” into numerical vectors. These vectorized biological sequences are then used for predicting protein function, estimating molecular structure, and serving as input to other predictive models.¹⁹

Graphs and networks are another major application. If you have a social network, a citation network, or a network of molecular interactions, you can “walk” along the connections between nodes in a pattern that mimics reading a sentence, then feed those walks into a skip-gram model to generate node embeddings.²⁰ These node vectors can then predict missing links in the network, detect communities, or classify individual nodes. Extensions like Het-node2vec have adapted the approach for heterogeneous graphs with different types of nodes and edges, capturing semantic information alongside structural patterns.²¹

How Embeddings Line Up With the Brain

One of the more surprising lines of research has been the discovery that word embedding spaces share geometric features with the way the human brain organizes language. Neuroscientists can record brain activity while people read or listen to words, then compare the pattern of distances between brain responses to the pattern of distances between the corresponding word vectors. A study using brain recordings found significant similarity between neural data and multiple embedding models, beginning within about 250 milliseconds of a person seeing a word, suggesting that both systems are shaped by the same structural properties of language.²²

More recent work has taken this further. Researchers found that contextual embeddings from large language models can serve as an explicit numerical model of the shared meaning space humans use during natural conversation, aligning brain activity in both speakers and listeners to the same embedding coordinates.²³ Another study demonstrated that contextual embeddings capture the geometry of brain activity in a language-processing region of the brain better than static word embeddings do, and that the common geometric patterns are strong enough to predict brain responses to words the model had never been tested on.²⁴ This does not mean machines “think” the way we do, but it does suggest that both artificial and biological systems converge on similar solutions when faced with the problem of organizing meaning.

Making Embeddings Interpretable

A persistent complaint about word embeddings is that the individual numbers in a vector are meaningless to a human. Dimension 47 might be slightly positive for “cat” and slightly negative for “democracy,” but there is no way to look at that number and understand what it encodes. The representations are powerful but opaque.

One approach to fixing this uses sparse coding: transforming the dense, uninterpretable vectors into a space where most values are zero and the few nonzero entries correspond to recognizable concepts. The idea is to decompose each word vector into a combination of a small number of “basis” concepts, so you can point to the active dimensions and say, roughly, what semantic features are contributing.²⁵ The tradeoff is that making embeddings more interpretable can sometimes reduce their performance on downstream tasks, so practitioners tend to use dense vectors for building systems and sparse variants for analysis and debugging.

Evaluating Whether Embeddings Are Any Good

There is no single test that tells you whether one set of embeddings is better than another, partly because “better” depends on what you plan to do with them. Evaluation approaches broadly split into intrinsic methods, which test properties of the vectors themselves, and extrinsic methods, which measure how well the embeddings perform when plugged into a real-world task. A comprehensive survey cataloged 16 intrinsic and 12 extrinsic approaches, underscoring how fragmented the evaluation landscape is.²⁶

Intrinsic tests include word similarity benchmarks (do the vectors rate “happy” and “joyful” as close together?) and the analogy tests described earlier. Extrinsic tests measure performance on actual tasks like sentiment analysis, named-entity recognition, or machine translation. The catch is that embeddings can score well on intrinsic tests and poorly on extrinsic ones, or vice versa. A set of vectors that perfectly captures word similarity might still underperform in a specific application if that application depends on syntactic rather than semantic relationships. This disconnect is a practical headache for engineers choosing which embeddings to use, and it is why most teams today evaluate directly on their target task rather than relying on generic benchmarks.