State of the art in machine learning, commonly abbreviated SOTA, refers to the best-performing methods on a given task at a given moment. In practice, the term has become shorthand for a fast-moving frontier where architectural breakthroughs, scaling strategies, and training recipes interact in ways that can make last year’s champion look outdated within months. As of mid-2025, SOTA is being reshaped by forces on several fronts simultaneously: new model architectures that challenge the Transformer’s dominance, a rethinking of how training compute should be balanced against inference compute, the looming exhaustion of high-quality web data, and a narrowing gap between open-weight and proprietary models.
Architectures Beyond the Standard Transformer
The Transformer architecture has dominated deep learning since 2017, but its quadratic cost on long sequences has fueled a steady search for alternatives. Approaches like linear attention, gated convolution, recurrent models, and structured state space models have all aimed to bring down that cost, though historically none matched Transformers on language tasks.1arXiv. Mamba: Linear-Time Sequence Modeling with Selective State Spaces The Mamba family of models introduced a selective state space mechanism that processes sequences in linear time and has shown competitive results on language modeling, pushing the conversation from “can anything replace attention?” to “when should we replace attention?”
Alongside alternatives to attention itself, the Mixture-of-Experts (MoE) paradigm has become one of the most consequential architectural trends. The core idea is that a model can have far more total parameters than it activates for any single input: a routing network selects a small subset of “expert” modules for each token, so the computational cost stays manageable even as the model’s total capacity grows enormously.2arXiv. The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications This sparse conditional computation is not new; the concept is roughly three decades old. But its recent re-emergence, in forms like Switch Transformers and related designs, has made it a standard tool for building extremely large but efficient models.3arXiv. A Review of Sparse Expert Models in Deep Learning Many of the top-performing open and closed language models now use some form of MoE under the hood.
How Scaling Laws Guide Training Decisions
One of the most influential ideas in recent ML is that the performance of a language model is predictable from how much compute you spend training it and how you split that budget between model size and data. The Chinchilla research demonstrated that many large language models were being undertrained: labs were pouring compute into bigger architectures while holding training data roughly constant. By training over 400 models of varying sizes, the researchers found that model size and training tokens should be scaled equally. Double your model, double your data.4arXiv. Training Compute-Optimal Large Language Models
That finding reshaped how labs plan training runs, but it assumed training cost was all that mattered. In reality, a model that will serve billions of inference requests has a very different cost profile than one trained for a benchmark paper. More recent work has adjusted the Chinchilla framework to account for inference demand, concluding that researchers expecting large-scale deployment (on the order of a billion requests) should train models that are smaller and trained for longer than the original Chinchilla-optimal prescription would suggest.5arXiv Central. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws The logic is straightforward: a slightly smaller model that performs nearly as well costs less on every single inference call, and over billions of calls, that savings dwarfs the extra training cost.
Test-Time Compute and the Reasoning Shift
Traditionally, making a model smarter meant spending more compute during training. A newer approach flips that: spend more compute at inference time, letting the model “think harder” about difficult prompts. Research on scaling test-time compute has found that, depending on the prompt’s difficulty, allocating extra computation during inference can be more effective than simply using a bigger model.6arXiv. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Two main mechanisms drive this. The first uses a verifier model to score candidate answers, searching through multiple completions to find better ones. The second adapts the model’s output distribution on the fly for each prompt. Both work, but their relative effectiveness depends on how hard the problem is, which has motivated “compute-optimal” strategies that allocate test-time resources adaptively rather than uniformly. This paradigm is behind the chain-of-thought and extended-reasoning capabilities in recent frontier models, where the model generates intermediate steps before producing a final answer. For math, coding, and logic problems, the gains from test-time scaling can be dramatic.
Making Inference Affordable Through Quantization
Even with the right model size and architecture, running large models in production is expensive. Quantization, which reduces the numerical precision of a model’s weights and activations, has become a critical tool for affordable deployment. A large-scale investigation spanning over 500,000 individual evaluations found that reducing precision from the standard 16-bit format to 8-bit floating point (FP8) is essentially lossless across all model sizes tested. Dropping to 8-bit integer precision, when properly calibrated, costs only about 1-3% in accuracy. And 4-bit integer weight-only quantization performs comparably to 8-bit integer schemes in many settings.7Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. “Give Me BF16 or Give Me Death”? Accuracy-Performance Trade-Offs in LLM Quantization
These findings matter because the difference between 16-bit and 4-bit means roughly a fourfold reduction in memory footprint, which translates directly to cheaper hardware requirements, faster inference, and the ability to run models on devices that could not have handled the full-precision version. Quantization is not a niche optimization anymore; it is a default step in any serious deployment pipeline.
The Data Wall and Synthetic Data
The scaling-law playbook assumes you can always get more training data by doubling it. In practice, high-quality web text is a finite resource, and evidence suggests that simply adding more of it eventually hits diminishing returns. This “data wall” has pushed researchers toward synthetic data: text generated by existing models and used to train new ones.8arXiv. BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
Synthetic data is promising but raises its own questions. It remains unclear whether synthetic datasets show the same predictable scaling behavior as natural web data.9arXiv. Scaling Laws of Synthetic Data for Language Models Naive approaches to model-generated training data risk “model collapse,” where biases and errors compound across generations. The research community is actively working on understanding when and how synthetic data helps, what quality filters are needed, and whether there are fundamental limits to bootstrapping intelligence from a model’s own outputs. For now, the most successful recipes blend curated web data with targeted synthetic data for domains like math and code.
Aligning Models to Human Preferences
Training a large model on text gives it raw capability, but making it actually useful and safe requires a second phase often called alignment. The standard approach, reinforcement learning from human feedback (RLHF), works but involves fitting a separate reward model, sampling from the language model during training, and navigating an unstable optimization process. Direct Preference Optimization (DPO) emerged as a simpler alternative that reformulates the same objective as a straightforward classification problem on human preference data, eliminating the need for a separate reward model or reinforcement learning loop entirely.10NeurIPS Proceedings. Direct Preference Optimization: Your Language Model is Secretly a Reward Model
DPO and its variants have rapidly become a default post-training technique because they are lighter, more stable, and require less hyperparameter fiddling than full RLHF. The practical impact is significant: alignment is no longer something only well-resourced labs can do well. Open-source projects now routinely apply DPO-style training to base models, and the results are competitive. The Tulu 3 family, for instance, used a fully open post-training pipeline built on Llama 3.1 base models and achieved results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models like GPT-4o-mini and Claude 3.5-Haiku.11arXiv. Tulu 3: Pushing Frontiers in Open Language Model Post-Training
Open-Weight Models Closing the Gap
For years, proprietary models from a handful of companies defined the SOTA ceiling. That gap has narrowed substantially. The success of Tulu 3 is one data point; more broadly, open-weight model families like Llama, Mistral, and Qwen now trade benchmark positions with closed models on many tasks. The shift has been driven not just by base model improvements but by better post-training recipes, openly shared datasets, and community-driven fine-tuning pipelines that let smaller teams iterate quickly.
The practical implication for anyone choosing a model is that the closed-vs-open decision is no longer primarily about raw quality. It is about factors like data privacy, deployment flexibility, customization needs, and vendor dependency. For many applications, an open model fine-tuned on domain-specific data outperforms a general-purpose proprietary model, and it does so at a fraction of the per-query cost since you control the hardware.
Multimodal Models and Native Integration
SOTA models increasingly work with more than text. The first generation of multimodal systems bolted separate vision encoders onto frozen language model backbones, a late-fusion approach that could handle images and text but treated the modalities as somewhat separate streams. The frontier has now moved toward native multimodal modeling, where different modalities are integrated from the ground up during training rather than glued together afterward.12arXiv. Toward Native Multimodal Modeling: A Roadmap
Video generation has seen particularly rapid progress. Latent diffusion transformers, which model video distributions in a compressed latent space using Transformer blocks to capture both spatial and temporal patterns, have achieved strong results across standard benchmarks.13arXiv. Latte: Latent Diffusion Transformer for Video Generation The underlying principle mirrors what happened with image generation a few years earlier: once someone figures out how to train Transformers effectively in a compressed representation space, quality jumps quickly.
Long Context Windows Versus Retrieval
Models with context windows stretching to hundreds of thousands of tokens have prompted a natural question: do you still need retrieval-augmented generation (RAG) if the model can just read everything at once? The evidence so far suggests yes, you probably do. In benchmark evaluations designed to test needle-in-a-haystack retrieval across long documents, RAG achieved a win rate of about 83% over models using direct long-context answers.14ACM Transactions on Information Systems. U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack And on more complex tasks requiring summarization across a large document set, even top models like GPT-4o and Claude 3 Opus scored below 20% without retrieval.15ACL Anthology. Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
Long context is useful for tasks where you need the model to reason over an entire document it has fully in view, like editing a codebase or reviewing a legal contract. But for tasks that require locating specific information across a large corpus, retrieval still wins. The two approaches are more complementary than competitive, and most production systems use both.
Agents, Tool Use, and Code as an Interface
One of the more consequential SOTA trends is using language models as autonomous agents that can plan multi-step tasks, call external tools, and act on the results. Code turns out to be a surprisingly powerful lever here. Beyond code generation per se, the structured nature of programming languages helps models produce precise intermediate steps, connect to external tools through function calls, and benefit from execution feedback when code runs and either succeeds or throws an error.16arXiv. If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents
The agentic paradigm has moved rapidly from research demos to production systems for tasks like data analysis, customer support workflows, and software development. The gap between what agents can do in a sandboxed evaluation and what they can do reliably in the real world remains significant, though. Error propagation across long action chains, the challenge of knowing when to ask for human input, and the risks of tool misuse are all active research problems.
Interpretability and Mechanistic Understanding
As models grow more capable, understanding what they are actually doing internally becomes both harder and more important. A major obstacle is polysemanticity: individual neurons in a neural network activate in multiple, seemingly unrelated contexts, making them difficult to interpret. Sparse autoencoders have emerged as a promising tool for decomposing model activations into more interpretable features, extracting sets of sparsely activating directions that correspond to recognizable concepts.17arXiv. Sparse Autoencoders Find Highly Interpretable Features in Language Models Early work demonstrated that this technique could extract interpretable features from small one-layer transformers,18Transformer Circuits Thread. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning and subsequent research has scaled the approach to larger models.
Interpretability work matters for SOTA in two ways. First, it feeds directly into safety: if you can identify where a model stores dangerous knowledge, you can attempt to remove or modify it. Second, it is starting to guide architecture and training decisions, since understanding which features emerge under different training conditions helps researchers build better models, not just explain existing ones.
Benchmark Saturation and the Contamination Problem
If you have followed ML benchmarks, you have probably noticed that models now routinely achieve very high scores on tests that seemed hard just a year or two ago. Part of this reflects genuine improvement. But part of it reflects a subtler problem: data contamination, where benchmark data ends up in a model’s training set, inflating scores without corresponding real-world capability. This has driven a shift toward dynamic evaluation methods that continuously update test data or regenerate questions to stay ahead of training contamination.19arXiv. Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation
The contamination issue makes it harder than ever to know what SOTA really means on a given benchmark. A score that looks like a breakthrough could partly reflect memorization of test examples rather than generalized capability. This is why the field is increasingly interested in evaluations that test models on genuinely novel problems, adversarial setups, and real-world tasks where memorization cannot help.
Running Models on Phones and Edge Devices
SOTA is not only about the biggest model on the biggest cluster. A parallel track focuses on getting useful models to run on resource-constrained devices like smartphones, embedded systems, and IoT hardware. Techniques combining parameter-efficient fine-tuning with structured pruning allow substantial language models to run on devices that would choke on the full-size version.20arXiv. Efficient Deployment of Large Language Models on Resource-constrained Devices Federated learning frameworks can distribute the fine-tuning process across many devices without centralizing user data, addressing privacy concerns that matter in healthcare, finance, and personal assistant applications.
On-device inference also sidesteps the latency and connectivity requirements of cloud-based models. For applications that need instant responses or must work offline, edge deployment is not a compromise; it is a requirement. The quantization techniques discussed earlier are central to this effort, since a 4-bit model that fits in a phone’s memory while maintaining most of its accuracy is exactly what edge deployment needs.
Energy, Water, and Environmental Cost
The environmental footprint of training and running large models generates heated debate, and the picture is more nuanced than either the “AI is destroying the planet” or “it’s fine” camps suggest. A lifecycle analysis comparing large language models to a human performing the same writing tasks found that on a per-page basis, models consumed dramatically less energy, emitted less carbon, and used less water than the human baseline. For a large model like Llama-3-70B, the human-to-model ratios ranged from about 40 to 150 across energy, carbon, water, and cost. For a smaller model like Gemma-2B, the ratios were even more extreme, ranging from roughly 1,200 to 4,400.21Scientific Reports. Reconciling the contrasting narratives on the environmental impact of large language models
The caveat is that this per-task framing obscures the aggregate picture. A human writing one page is one event. A model serving millions of users simultaneously is millions of events. Training a frontier model from scratch involves thousands of GPUs running for weeks. The per-query efficiency advantage is real, but the total resource draw of the AI industry is growing rapidly because demand is growing rapidly. Novel interconnect designs for distributed training, such as wafer-scale fabrics that reduce network traffic and improve end-to-end training time by roughly 1.3 to 1.9 times across various large models, help at the margins but do not change the fundamental trajectory.22arXiv. FRED: Flexible REduction-Distribution Interconnect and Communication Implementation for Wafer-Scale Distributed Training of DNN Models
Safety, Unlearning, and Adversarial Robustness
Frontier models are typically fine-tuned to refuse requests for dangerous content, but those guardrails are often brittle. Jailbreaks that bypass refusal training are discovered regularly, and the cat-and-mouse dynamic shows no sign of ending. Machine unlearning aims for something more thorough: actually removing hazardous knowledge from a model’s weights so that it cannot be extracted even by a determined adversary.23arXiv. An Adversarial Perspective on Machine Unlearning for AI Safety In practice, existing unlearning methods are not yet robust enough against adversarial attacks, making this an area where the gap between the aspiration and the reality is still wide.
Safety evaluation itself is evolving. Static red-teaming, where a fixed set of prompts is used to test for harmful outputs, faces the same contamination issues as benchmarks. Models can be trained to pass specific safety tests without developing genuinely safe behavior in novel situations. The result is a growing emphasis on adaptive, dynamic safety testing and on developing formal guarantees rather than empirical pass rates.
Legal Uncertainty Around Training Data
A question hanging over the entire SOTA landscape is whether the training data practices that produced current models are legal. Roughly fifty copyright lawsuits have been filed against AI companies, centering on whether using copyrighted works to train models constitutes fair use. The debate is deeply polarized: one side views it as straightforward theft from creators, the other as a legitimate technological fair use that produces tools with broad societal benefit. Both sides cite the same Supreme Court precedents to support their position.24Houston Law Review. Fair Use and the Origin of AI Training
The outcome of these cases could reshape what SOTA looks like going forward. If courts rule that training on copyrighted data requires licenses, the cost of assembling training datasets could rise dramatically, further concentrating frontier model development among well-capitalized companies. If training is found to be fair use, the current ecosystem continues largely intact. Either way, the legal uncertainty is one reason synthetic data and carefully licensed datasets have become active research areas, since they represent a path forward regardless of how the courts rule.