Sentence transformers are specialized models that convert entire sentences or short passages into fixed-length numerical representations, called embeddings, that capture their meaning. They emerged from a straightforward problem: earlier language models like BERT were powerful at understanding text, but painfully slow when you needed to compare thousands or millions of sentences to each other. The solution involved restructuring how these models process text so that each sentence gets its own standalone vector, making comparisons fast and scalable. That seemingly simple architectural shift has made sentence transformers the backbone of modern semantic search, recommendation systems, and a wide range of applications where machines need to judge how similar two pieces of text really are.
The Speed Problem That Started Everything
Before sentence transformers existed, the standard approach for measuring how similar two sentences were involved feeding both sentences into a model like BERT together, as a single combined input. The model would then produce a similarity score. That works well enough for a handful of comparisons, but it scales terribly. Finding the most similar pair in a collection of just 10,000 sentences required roughly 50 million inference computations, which took around 65 hours with BERT.1ACL Anthology. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks For anything involving real-time search over a large corpus, this was a non-starter.
The breakthrough came in 2019 with Sentence-BERT (SBERT), which modified BERT using a siamese network structure. Instead of processing two sentences together, SBERT encodes each sentence independently into its own vector. You can then compare any two sentences by simply measuring the distance between their vectors using cosine similarity, a calculation so lightweight it takes microseconds.2ACL Anthology. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks That same 10,000-sentence comparison task goes from 65 hours to a few seconds. The vectors can also be pre-computed and stored, so you only run the model once per sentence rather than once per comparison.
How Sentence Transformers Produce Embeddings
At its core, a sentence transformer takes a piece of text, runs it through a transformer-based language model, and then pools the output into a single vector of fixed length. That vector is supposed to live in a space where sentences with similar meanings end up close together and sentences with different meanings are far apart. The model learns this arrangement during training, typically by being shown pairs of sentences that are known to be semantically similar (or dissimilar) and adjusting its internal weights so the resulting vectors reflect those relationships.
This “encode once, compare cheaply” design is what the field calls a bi-encoder architecture. Each piece of text gets encoded independently, and the comparison happens entirely in the vector space. The alternative, where both texts are processed together with full cross-attention between them, is called a cross-encoder. Cross-encoders generally produce more accurate similarity judgments because the model can directly attend to relationships between the two texts, but they are far too slow for large-scale retrieval since every candidate pair requires a fresh pass through the model.3ACL Anthology. Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks A common compromise in practice is to use a sentence transformer as the first stage to quickly narrow a large set of candidates down to a shortlist, then re-rank that shortlist with a cross-encoder.
Contrastive Learning and Why It Changed the Game
The quality of sentence embeddings depends heavily on how the model is trained, and the biggest leap in quality came from contrastive learning. The idea is to train the model by pushing similar sentences closer together in vector space while pushing dissimilar sentences apart. Different approaches vary in how they define “similar” and “dissimilar” and what data they require.
SimCSE, introduced in 2021, demonstrated a particularly elegant unsupervised approach. It feeds the same sentence through the model twice, relying on the randomness of dropout (a standard regularization technique where some internal connections are randomly deactivated during each forward pass) to produce two slightly different representations of the same input. Those two representations serve as the “positive pair” that should be close together, while representations of other sentences in the batch serve as negatives. This method, despite requiring no labeled data, performed on par with earlier supervised methods.4ACL Anthology. SimCSE: Simple Contrastive Learning of Sentence Embeddings The supervised version of SimCSE pushed further by using sentence pairs from natural language inference datasets, treating sentences that entail each other as positives and sentences that contradict each other as hard negatives. The supervised variant achieved an average Spearman’s correlation of about 81.6% on semantic similarity benchmarks, a meaningful jump over previous methods.5ACL Anthology. SimCSE: Simple Contrastive Learning of Sentence Embeddings
Subsequent work has refined these ideas further. SimCSE++ identified that dropout noise in negative pairs was degrading performance and proposed methods to handle it, gaining nearly two points on top of SimCSE’s already strong baseline.6ACL Anthology. SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives Other work has combined supervised contrastive loss with standard cross-entropy loss during training, reporting gains of around 2.8% on similarity benchmarks over using cross-entropy alone.7arXiv. Sentence Embeddings using Supervised Contrastive Learning The trajectory is clear: contrastive objectives have become the default training recipe for sentence transformers, and incremental improvements keep accumulating.
Measuring What Actually Works
For a long time, sentence embedding models were evaluated on a narrow set of semantic similarity benchmarks, which gave a limited picture of how well they worked across different tasks. The Massive Text Embedding Benchmark (MTEB) was introduced to address this, covering eight distinct embedding tasks across 58 datasets and 112 languages.8ACL Anthology. MTEB: Massive Text Embedding Benchmark These tasks include retrieval, classification, clustering, pair classification, and reranking, among others.
One of the most informative findings from MTEB is that no single embedding model dominates across all tasks.9ACL Anthology. MTEB: Massive Text Embedding Benchmark A model that excels at retrieval might be mediocre at clustering, and vice versa. This matters practically: if you are building a search engine, you want to choose a model that scores well on retrieval tasks specifically, not one that has the highest average across all categories. MTEB has become the standard leaderboard for the field, and checking it is the fastest way to compare current models for your use case.
Making Embeddings Smaller Without Losing Meaning
A typical sentence transformer produces vectors with 768 or 1024 dimensions. When you are storing embeddings for millions or billions of documents, those dimensions add up fast. Storage costs, memory requirements, and the speed of nearest-neighbor search all scale with the size of each vector, creating pressure to shrink embeddings without destroying their usefulness.
Matryoshka Representation Learning (MRL) offers an especially clever solution. Named after Russian nesting dolls, it trains the model so that the first dimensions of an embedding carry the most important information, with each additional dimension adding finer detail. You can then truncate an embedding to whatever size your application can afford: use the full 768 dimensions when accuracy is critical, or just the first 128 or 64 when speed and storage matter more. This approach adds no extra cost during inference, and the truncated representations perform at least as well as models that were independently trained at those smaller dimensions.10NeurIPS Proceedings. Matryoshka Representation Learning Several popular embedding models now support Matryoshka-style truncation out of the box.
For even more aggressive compression, researchers have explored quantizing floating-point embeddings into binary vectors, where each dimension is reduced to a single bit. One approach using random hyperplane projections achieved storage reductions of up to about 99% while still preserving enough information for reasonable retrieval performance.11arXiv. Rediscovering Hashed Random Projections for Efficient Quantization of Contextualized Sentence Embeddings Binary vectors also enable extremely fast similarity computation using bitwise operations, which is useful for hardware-constrained environments.
Going Multilingual
Most sentence transformer research started in English, but extending these models to other languages has been a major area of development. The core technique is knowledge distillation: take a strong English-language sentence transformer as a “teacher,” gather parallel sentence pairs (the same sentence in English and a target language), and train a multilingual “student” model to produce the same embeddings for the translated sentences that the teacher produces for the English originals.12ACL Anthology. Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation The beauty of this approach is that it can extend any high-quality monolingual model to new languages without retraining it from scratch.
This distillation idea has been pushed further into domain-specific territory. Recent work has applied cross-lingual distillation to biomedical text classification, aligning sentence embeddings between an English biomedical teacher and a multilingual student model. The result transfers both domain knowledge and language coverage simultaneously.13Knowledge-Based Systems. Cross-lingual distillation for domain knowledge transfer with sentence transformers For organizations that need to process specialized documents in multiple languages, this is a practical shortcut that avoids training separate models for each language-domain combination.
When Large Language Models Become Sentence Encoders
The rise of large generative models like GPT and Llama has raised a natural question: can these massive decoder-only models also produce good sentence embeddings? The answer is yes, but it requires some adaptation. Decoder-only models are designed to predict the next word, not to compress an entire sentence’s meaning into a single vector. Their internal attention mechanism only looks backward (each word attends to previous words, not future ones), which limits their ability to represent full sentence meaning the way bidirectional models can.
LLM2Vec demonstrated a straightforward approach: enable bidirectional attention, add a masked prediction step, and apply unsupervised contrastive learning. These three modifications can turn a decoder-only model into a competitive text encoder.14arXiv. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders Causal2Vec takes a different philosophy, enhancing decoder-only models for embedding tasks without altering their architecture or attention patterns at all, preserving the semantic knowledge they acquired during pretraining.15ACL Anthology. Causal2Vec: Improving Decoder-only LLMs as Embedding Models through a Contextual Token
This trend is significant because it means the enormous investment in training large generative models can be repurposed for embedding tasks. You do not necessarily need a separate sentence transformer architecture if you can adapt the language model you already have. In practice, though, dedicated sentence transformers still tend to be faster and cheaper to run, so the choice depends on whether you prioritize quality at the frontier or efficiency at scale.
The Geometry Problem Lurking Inside Embeddings
Transformer-based embedding models have a known geometric quirk: their representations tend to be anisotropic, meaning the vectors cluster into a narrow cone in the high-dimensional space rather than spreading out evenly in all directions. This matters because it reduces the effective use of the available space and can make similarity scores less meaningful, since even unrelated sentences end up relatively close together.16arXiv. Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics Longer texts can exacerbate the problem, leading to what researchers call length-induced representation collapse, where texts of different lengths get mapped to similar regions regardless of content.17arXiv. Pooling and Semantic Shift: The Fundamental Challenges in Long Text Embedding and Retrieval
Contrastive learning, as discussed earlier, directly combats anisotropy by pushing the embedding space to be more uniform. The SimCSE paper showed both theoretically and experimentally that its contrastive objective regularizes the space toward uniformity.18ACL Anthology. SimCSE: Simple Contrastive Learning of Sentence Embeddings Interestingly, more recent research has found that anisotropy is not necessarily inherent to all transformers. Certain model families, specifically larger Pythia models, exhibit naturally isotropic embedding spaces, suggesting the problem may be tied to specific architectures and training setups rather than being a universal limitation.19ACL Anthology. Anisotropy is Not Inherent to Transformers
Where Sentence Transformers Still Struggle
Despite their impressive performance on standard benchmarks, sentence transformers have blind spots that are worth knowing about. One of the most stubborn is negation. Models tend to assign high similarity scores to sentence pairs where one sentence is the negation of the other. “The movie was good” and “The movie was not good” end up with embeddings that sit close together, because the words overlap almost entirely and the small negation word does not shift the vector enough. The position of a negation word in a sentence can sharply change its meaning, but embeddings often fail to capture this.20Nature. Computation of sentence similarity score through hybrid deep learning with a special focus on negation sentence If your application depends on distinguishing affirmative and negative versions of the same claim, this is a real limitation worth testing for.
Social bias is another concern. Sentence embeddings inherit biases from their training data, reflecting societal patterns around gender, race, and other demographic categories. These biases can propagate into downstream applications: a biased embedding model used for resume screening, for instance, might systematically rank certain groups lower. The Sent-Debias method has been shown to reduce these biases while preserving performance on downstream tasks like sentiment analysis and language understanding.21ACL Anthology. Towards Debiasing Sentence Representations Measuring bias itself is tricky, though, as most metrics rely on cosine similarity and may not capture all dimensions of the problem.22arXiv. Evaluating Metrics for Bias in Word Embeddings
What Do Sentence Embeddings Actually Encode
A single vector with 768 numbers somehow captures the meaning of a sentence. But what exactly is being captured, and what gets lost? Probing studies have tried to answer this by designing targeted tasks that test whether specific linguistic properties can be recovered from the embedding alone. A foundational study introduced ten probing tasks designed to capture simple linguistic features, examining how different encoders and training methods affect what information ends up in the vector.23arXiv. What you can cram into a single vector: Probing sentence embeddings for linguistic properties Follow-up work has extended this to sub-sentence structure, probing whether embeddings preserve information about syntax, named entities, semantic roles, and other structured properties that traditional NLP pipelines extract in separate stages.24arXiv. What do you learn from context? Probing for sentence structure in contextualized word representations
The general finding is that sentence embeddings encode a surprising amount of linguistic information, but unevenly. Surface-level features like sentence length and word presence are easy to extract. Deeper properties like tree depth or semantic role assignments are captured to varying degrees depending on how the model was trained. This uneven encoding helps explain why models perform well on some tasks and poorly on others, and why the MTEB leaderboard shows no single model dominating everywhere. If your task depends on a linguistic property that the model’s training did not emphasize, performance will suffer even if the model looks strong on aggregate benchmarks.
Adapting to Specialized Domains
General-purpose sentence transformers are trained on broad web text and commonly used datasets, which means they understand everyday language well but can stumble on specialized vocabulary and concepts. Medical texts, legal documents, aviation reports, and scientific papers all use language in domain-specific ways that a general model may not fully grasp. One emerging approach uses large language models with a few example prompts to generate domain-specific training queries, which are then used to fine-tune a retrieval model for that particular domain.25arXiv. Study on LLMs for Promptagator-Style Dense Retriever Training This dramatically reduces the need for expensive manually labeled training data in specialized fields.
The practical takeaway for anyone deploying sentence transformers in a niche domain: do not assume the off-the-shelf model will work well enough. Test it on examples from your actual data, and plan for at least a lightweight fine-tuning step. The gap between general-purpose and domain-tuned performance can be substantial, especially when your texts contain terminology or sentence structures that rarely appear in the model’s original training data.
Beyond Text Alone
Sentence transformers originally dealt exclusively with text, but the embedding paradigm naturally extends to other types of data. Multimodal embedding models aim to map both text and images into a shared vector space, so you can search for images using text queries or find text that matches a given image. Recent work has shown that training a model to produce good multimodal embeddings and to generate text descriptions of images (captioning) do not significantly conflict with each other, meaning a single model can serve both purposes.26arXiv. Multi-Modal Generative Embedding Model The Matryoshka approach described earlier has also been adapted for recommendation systems, structuring user and item representations in a nested hierarchy that captures preferences at different levels of granularity.27arXiv. Matryoshka Representation Learning for Recommendation The shared theme across these extensions is that the “encode once, compare cheaply” idea has legs well beyond sentence-level text similarity, and the infrastructure built around sentence transformers is increasingly being reused for data types the original creators never envisioned.