ChatGPT works by predicting the next word (technically, the next token) in a sequence, over and over, until it has built a complete response. Underneath the conversational interface is a large language model built on the transformer architecture, which was trained on enormous amounts of text from the internet and then further refined to follow instructions and behave in ways that feel helpful rather than random. The result is a system that does not “understand” language in the way you do, but produces text that is often remarkably coherent, and sometimes remarkably wrong, for reasons that are worth understanding.
Breaking Text Into Tokens
Before ChatGPT can do anything with your message, it needs to convert your words into numbers. It does this through tokenization, which chops text into small pieces called tokens. A token might be a whole word like “dog,” a piece of a word like “un” and “believe” and “able,” or even a single character. The specific method OpenAI uses is called byte pair encoding (BPE), which works by repeatedly merging the most common pairs of characters or character groups in a large training set until you get a vocabulary of manageable size. Research on tokenization has shown that the choice of tokenizer matters a lot, particularly for languages with different scripts or lower-resource languages where standard BPE vocabularies can be a poor fit.1Procedia Computer Science. Tokenization Matters: The Impact of Byte and BPE Encoder-Only Language Models on Low-Resource Adaptation in Multilingual Neural Machine Translation Each token gets mapped to a numerical vector, an embedding, that encodes its meaning in a high-dimensional space. These embeddings are what the model actually operates on.
Self-Attention and the Transformer
The core innovation behind ChatGPT is the transformer architecture, and the key idea inside a transformer is self-attention. In simple terms, self-attention lets every token in your input “look at” every other token and figure out which ones matter most for understanding the current word in context. The word “bank” means something very different in “river bank” versus “bank account,” and self-attention is the mechanism that resolves that ambiguity by weighing the surrounding words.
Mechanically, self-attention works by having each token generate three things: a query (what am I looking for?), a key (what do I contain?), and a value (what information do I carry?). The model computes how well each token’s query matches every other token’s key, producing a set of attention scores. Those scores act as weights: the model uses them to create a blended representation for each position that draws more heavily from the most relevant tokens.2Knowledge-Based Systems. Self-attention Mechanism at the Token Level: Gradient Analysis and Algorithm Optimization This happens many times in parallel across different “attention heads,” each of which can specialize in a different kind of relationship, like one head tracking grammar and another tracking meaning.
ChatGPT stacks dozens of these transformer layers on top of each other. The information passes through each layer, and each layer refines the representation further. Research has found that each layer in a large language model contributes roughly equally to improving prediction accuracy, from the first layer all the way to the last, which means no single layer does the heavy lifting alone.3Physical Review. A law of next-token prediction in large language models
Pre-Training on Massive Text
The foundation of ChatGPT’s abilities comes from pre-training, which is conceptually straightforward: give the model an enormous body of text, hide the next word, and ask it to guess what comes next. When it guesses wrong, adjust its internal parameters slightly to make a better guess next time. Repeat this billions of times across a dataset spanning books, websites, forums, code repositories, and more. Through this process, the model develops internal representations that capture grammar, facts, reasoning patterns, and stylistic tendencies, all as a side effect of getting better at next-word prediction.
The model never sees a lesson plan or a textbook labeled “this is how English grammar works.” It just learns, through statistical patterns in vast quantities of text, that certain words and structures tend to follow others in certain contexts. The sheer scale of the training data and the model’s parameter count is what allows it to pick up on so many different kinds of pattern.
Why Bigger Models Work Better
One of the most important findings in language model research is that performance improves in a remarkably predictable way as you scale up. Researchers at OpenAI demonstrated that the loss (a measure of how wrong the model’s predictions are) decreases as a power law with model size, dataset size, and the amount of computing power used for training, with these trends holding across more than seven orders of magnitude.4arXiv. Scaling Laws for Neural Language Models In practical terms, this means that if you double the number of parameters in a model and give it proportionally more data to train on, you can predict with reasonable confidence how much the model will improve.
Later work refined these scaling laws considerably. A landmark study by researchers at DeepMind found that most large language models had been significantly undertrained: they had too many parameters relative to the amount of training data. For compute-optimal training, the model size and the number of training tokens should be scaled equally, meaning for every doubling of model size, you should also double the training data.5arXiv. Training Compute-Optimal Large Language Models When earlier scaling law predictions seemed to disagree with this finding, researchers traced the discrepancy to technical details like how computational cost was measured and how the optimizer was tuned at different scales.6NeurIPS Proceedings. Scaling Laws for Neural Language Models – Reconciliation These scaling laws are a big reason why the AI industry has invested so heavily in ever-larger training runs.
Turning a Text Predictor Into a Helpful Assistant
A pre-trained language model is not yet ChatGPT. It can complete text, but it has no built-in motivation to be helpful, truthful, or safe. If you type a question, it might respond with a continuation that sounds like a Wikipedia article, or it might generate something that looks like a forum post, or it might just keep writing as if your question were the opening of a novel. The model needs further training to learn to act like an assistant, and this happens in two major stages.
The first stage is instruction tuning: the model is trained on carefully curated examples of instructions paired with high-quality responses. This teaches the model the basic pattern of “when someone asks a question, provide a direct helpful answer.” The effect is striking: even without changing the model’s fundamental capabilities, instruction tuning bridges the gap between a model trained to predict the next word and one that follows human instructions.7arXiv. Instruction Tuning for Large Language Models: A Survey Research has shown that instruction-following data can even be generated by advanced models themselves, bootstrapping the process without requiring all examples to be written by humans.8arXiv. Instruction Tuning with GPT-4
The second stage is alignment, which aims to make the model not just competent but also helpful, honest, and harmless. The best-known approach is reinforcement learning from human feedback (RLHF). In RLHF, human raters compare pairs of model responses and indicate which one they prefer. These preferences are used to train a separate reward model that scores how “good” a response is. Then the language model is further trained to maximize the reward model’s scores, nudging it toward the kinds of answers humans rated highly.9arXiv. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
RLHF is powerful but complicated. Helpfulness and harmlessness can pull in opposite directions: a maximally helpful model might provide dangerous information, while a maximally cautious model might refuse reasonable requests. Some researchers have proposed separating these objectives explicitly, training distinct models for helpfulness and safety and then optimizing both simultaneously under constraints.10arXiv. Safe RLHF: Safe Reinforcement Learning from Human Feedback Others have questioned whether RLHF can truly align an AI system with human values at all, given the difficulty of representing diverse and sometimes contradictory human preferences through simple pairwise comparisons.11arXiv. AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations
A newer alternative called Direct Preference Optimization (DPO) simplifies the whole process. Instead of training a separate reward model and then running reinforcement learning, DPO folds both steps into a single training procedure that directly adjusts the language model’s behavior based on human preference data. It is computationally lighter and more stable than traditional RLHF, and it eliminates the need to sample from the model during fine-tuning.12NeurIPS Proceedings. Direct Preference Optimization: Your Language Model is Secretly a Reward Model DPO has rapidly become a popular alternative in the field, with numerous variants and applications explored in recent years.13arXiv. A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
How It Generates Each Response
When you send ChatGPT a message, it does not retrieve a pre-written answer from a database. It generates text one token at a time. At each step, the model produces a probability distribution over its entire vocabulary: every possible next token gets a score indicating how likely it is to come next given everything that preceded it. The model then selects a token from this distribution and appends it to the sequence, repeating the process until it decides the response is complete.
How the model picks from that distribution matters more than you might think. The simplest approach, greedy decoding, just picks the single most probable token every time, but this tends to produce repetitive and dull text. More commonly, the system uses sampling strategies that introduce controlled randomness. Top-k sampling limits choices to the k most probable tokens. Top-p (nucleus) sampling includes tokens until their cumulative probability reaches some threshold. Recent research has framed all of these decoding methods as special cases of a single optimization problem over the probability distribution, showing that the right sampling strategy can substantially improve accuracy on tasks like math.14arXiv. Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers A “temperature” parameter controls how adventurous the sampling is: low temperature makes the model more conservative and predictable, high temperature makes it more creative and unpredictable.
The Context Window and Its Limits
ChatGPT does not have a persistent memory across conversations (at least not by default). Everything it “knows” about your current conversation comes from the context window: the sequence of tokens it can see at once, encompassing your messages, its own prior replies, and any system instructions. Early GPT models had context windows of a few thousand tokens. More recent models can handle tens or even hundreds of thousands of tokens, but there are still hard limits.
The model keeps track of the conversation using a key-value (KV) cache, which stores the computed attention information for all tokens generated so far, avoiding the need to recompute everything from scratch at each step. Research has shown that when this cache approaches or exceeds the model’s trained context window, output quality degrades sharply, and not just because the system runs out of memory. The transformer relies on positional encodings to keep track of where each token sits in the sequence, and strategies that try to compress or evict old tokens from the cache can scramble those positional signals and produce incoherent outputs.15arXiv. Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity
Various techniques for positional encoding have been developed to help transformers handle longer contexts, including methods like Rotary Position Embeddings (RoPE) that encode position information directly into the attention computation.16arXiv. Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling Newer approaches are exploring ways to decouple positional information from the cached data entirely, which would allow the model to reuse cached computations more flexibly across different contexts.17arXiv. LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding For you as a user, the practical takeaway is that very long conversations will eventually hit a wall where the model either loses track of earlier content or starts producing lower-quality responses.
Why It Makes Things Up
One of the most discussed limitations of ChatGPT is its tendency to generate confident-sounding statements that are simply false. These are often called “hallucinations,” though some researchers have argued that “fabrications” or “falsifications” are more accurate terms, since the model is not perceiving something that is not there; it is constructing plausible-sounding text based on probabilistic patterns. The cause has been linked to the fundamental nature of how these models produce language: they generate outputs based on estimates of what words are most likely to come next, which allows bits of false information to be mixed seamlessly with factual content.18Schizophrenia. ChatGPT: these are not hallucinations – they’re fabrications and falsifications
This happens because the model has no internal fact-checker. It does not look up information in a database or verify its claims against a source of truth. It generates text that is statistically plausible given its training data and the conversation context. When the training data contains conflicting information, or when the question requires precise knowledge the model has only loosely encoded, it fills in the gaps with plausible-sounding inventions. This is why ChatGPT can confidently cite academic papers that do not exist or describe historical events that never happened: the model is optimizing for “sounds right” rather than “is right.”
Safety Guardrails and Their Weak Points
OpenAI and other providers have built extensive safety systems around their models, often called guardrails, to prevent the generation of harmful, biased, or dangerous content. These include content filters, system-level instructions, and the alignment training discussed earlier. Developing robust safeguards has become a major area of research as these models have become more widely deployed.19PubMed Central. Safeguarding large language models: a survey
These guardrails are not foolproof. Red-teaming research, where security researchers deliberately try to break the safety systems, has shown that ChatGPT’s protections can be bypassed through carefully crafted prompts. While the model performs well on standard benchmark safety tests, manually curated adversarial examples have exposed risks across several categories, including social bias in generated code, toxic language in personalized dialogue, misleading information from fabricated content, and vulnerability to prompt injection attacks that manipulate the model into ignoring its safety instructions.20ACM Transactions on Software Engineering and Methodology. Bypassing Guardrails: Lessons Learned from Red Teaming ChatGPT This is an ongoing arms race: providers patch vulnerabilities, and researchers find new ones.
Retrieval-Augmented Generation and External Tools
One way to address the fabrication problem and keep ChatGPT’s knowledge current is retrieval-augmented generation (RAG). Instead of relying entirely on what the model memorized during training, RAG systems retrieve relevant documents from an external database at the time you ask your question and feed that information into the model’s context window. The model then generates its answer grounded in the retrieved evidence rather than relying purely on its internal parameters.
RAG has evolved rapidly from simple retrieve-then-generate pipelines into a broad design space that includes different retrieval methods, multi-stage document reranking, evidence fusion strategies, and even agentic workflows where the model decides what to search for and when.21Wiley Online Library. Retrieval‐Augmented Generation for Large Language Models: Evolution, Architectures, Applications, and Challenges (2020–2025) When you use ChatGPT with browsing enabled, or when it accesses uploaded files, you are seeing a form of RAG in action. The model’s responses become more accurate and verifiable because it can point to specific sources rather than generating from memory alone.
Mixture of Experts and Efficient Scaling
Training and running ever-larger language models gets expensive fast, which has driven interest in more efficient architectures. One prominent approach is the Mixture of Experts (MoE) design, where instead of running every input through every parameter in the model, the system routes each token through only a subset of specialized “expert” sub-networks. This allows the total model to have a very large number of parameters, giving it enormous capacity, while keeping the computation for any individual token much smaller than it would be in a standard dense model.22arXiv. Mixture of Experts in Large Language Models
The sparsity in MoE models is the key advantage: the parameter count can grow without a proportional increase in computational cost, because most parameters are inactive for any given input.23arXiv. Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity While OpenAI has not fully disclosed ChatGPT’s architecture, reporting strongly suggests that recent GPT models use some form of MoE, which would explain how they can offer strong performance without the prohibitive inference costs of a fully dense model at the same scale.
The Energy Footprint
Every time you ask ChatGPT a question, servers in a data center run billions of calculations to generate the response. Multiply that by hundreds of millions of users and the energy costs add up fast. Training a large language model from scratch consumes enormous amounts of electricity, and the ongoing costs of inference, fine-tuning, and updating are substantial as well. Researchers have pointed out that the entire life cycle of an LLM-powered chatbot, from data collection and training to deployment and continuous updates, has significant energy and carbon footprint implications that deserve attention as these systems become more widespread.24Engineering. Preventing the Immense Increase in the Life-Cycle Energy and Carbon Footprints of LLM-Powered Intelligent Chatbots Efficiency improvements like MoE architectures, better hardware utilization, and smarter caching strategies help, but the fundamental trend has been toward larger models, longer contexts, and more users, all pushing energy consumption upward.
The environmental dimension is one reason why research into compute-optimal training has practical stakes beyond just model performance. If you can get the same quality from a smaller model trained on more data, or from a sparse architecture that activates only a fraction of its parameters, the electricity savings at the scale of a globally deployed chatbot are enormous. Whether the industry will prioritize those savings over pushing toward ever-more-capable models remains an open question.