Speech synthesis converts written text into spoken audio through a series of stages: analyzing the text, generating an acoustic representation, and producing a final waveform. Modern systems use neural networks for most of this work, but the fundamental pipeline has remained consistent for years. Text goes in as characters, gets transformed into linguistic features, then into a visual map of sound frequencies called a spectrogram, and finally into the raw audio samples that play through your speakers.
The Three-Stage Pipeline
Every text-to-speech system, whether it’s reading your GPS directions or narrating an audiobook, follows three core stages: text analysis, acoustic modeling, and vocoding. This design mirrors how human speech actually works. Your vocal cords produce a raw sound (the source), and your mouth, tongue, and throat shape that sound into recognizable speech (the filter). TTS systems split the job the same way: one component figures out what to say and how to say it, another generates a blueprint of the sound, and a third builds the actual audio.
Separating these stages lets engineers optimize each one independently. The text analysis component focuses on understanding language. The acoustic model focuses on producing natural-sounding patterns. The vocoder focuses on audio fidelity. This modularity is why you can swap in a different voice or language without redesigning the entire system.
Text Analysis: Turning Words Into Instructions
Raw text is full of ambiguity that humans resolve effortlessly but computers cannot. The text analysis stage, sometimes called the frontend, cleans up the input and converts it into a precise set of pronunciation instructions. This happens in several steps.
First, the system normalizes non-standard text. The string “Dr. Smith lives at 123 Oak St.” needs to become “Doctor Smith lives at one twenty-three Oak Street.” Dates, currency symbols, abbreviations, and numbers all get expanded into their spoken forms. This sounds simple, but consider that “Dr.” could mean “Doctor” or “Drive” depending on context, and “1/2” might be “one half,” “January second,” or “one slash two.”
Next comes phonetic conversion: mapping each word to its actual pronunciation. The system first checks a pronunciation dictionary. If a word isn’t found (a brand name, a foreign word, a newly coined term), a separate module called a grapheme-to-phoneme converter predicts the pronunciation based on spelling patterns. This prediction step is error-prone, which is why TTS systems occasionally mangle unusual names.
Homographs present a particular challenge. The word “read” is pronounced differently in “I read books” versus “I read that yesterday.” The word “lead” changes depending on whether it’s a verb or a metal. The frontend uses context clues, often through part-of-speech tagging, to pick the right pronunciation. Modern systems handle common homographs well but can still stumble on rare or ambiguous cases.
Finally, the frontend predicts prosodic structure: where pauses should fall, which words deserve emphasis, and where the pitch should rise or drop. Clause boundaries, commas, and question marks all signal changes in rhythm and intonation. The output of this entire stage is a sequence of phonemes (individual speech sounds) annotated with stress and phrasing information.
Acoustic Modeling: Creating a Sound Blueprint
The acoustic model is the core intelligence of the system. It takes the phoneme sequence from the frontend and generates a mel spectrogram, which is essentially a detailed visual map of how the sound’s energy should be distributed across different frequencies over time. The “mel” part refers to a frequency scale designed to match how human ears perceive pitch. We’re more sensitive to differences between low frequencies than high ones, so the mel scale compresses the upper range accordingly.
Why not skip the spectrogram and predict audio samples directly? Because audio requires modeling tens of thousands of individual samples every second. A spectrogram compresses that information dramatically, capturing the essential shape of the sound without the computational burden of generating every single sample point. The acoustic model can focus on getting the linguistic content right (pronunciation, rhythm, emotion) while leaving the fine details of audio quality to the next stage.
Older systems used recurrent neural networks that processed text one step at a time, which was slow. Modern systems use transformer architectures, the same technology behind large language models. Transformers process the entire input sequence in parallel using a mechanism called multi-head attention, which lets the model look at all parts of the sentence simultaneously to understand context. This dramatically speeds up training and improves the model’s ability to handle long sentences with complex structure.
How Prosody Gets Encoded
The difference between robotic and natural-sounding speech comes down to prosody: the melody and rhythm layered on top of the words. Two acoustic properties matter most. The first is fundamental frequency, which corresponds to pitch. Generating a natural pitch contour from text means knowing that questions rise at the end, that emphasized words jump in pitch, and that speakers don’t maintain a flat tone across a sentence. The second is timing. Stressed syllables are longer than unstressed ones, and syllables just before a pause stretch out noticeably. Getting these patterns wrong produces speech that is technically intelligible but sounds uncanny.
Neural acoustic models learn these patterns implicitly from thousands of hours of recorded human speech. They don’t follow explicit rules about where pitch should rise or fall. Instead, they absorb statistical patterns from training data, which is why a model trained on audiobook narration sounds different from one trained on conversational speech.
Vocoding: Building the Actual Audio
The vocoder takes the mel spectrogram and synthesizes a raw audio waveform from it. Think of the spectrogram as a detailed architectural blueprint and the vocoder as the construction crew that builds the actual structure. The spectrogram tells the vocoder what frequencies should be present at each moment, and the vocoder fills in the fine-grained details needed to produce a continuous stream of audio samples.
Early neural vocoders like WaveNet, developed by DeepMind, produced remarkably natural audio but were painfully slow because they generated one audio sample at a time, and speech requires 16,000 to 24,000 samples per second. Newer vocoders like HiFi-GAN use a different approach: they generate large chunks of the waveform at once, achieving real-time or faster-than-real-time speeds without sacrificing much quality. This is what made neural TTS practical for consumer products like voice assistants and navigation apps.
End-to-End Systems
The traditional three-stage pipeline works well, but each handoff between stages can introduce errors. If the frontend mispredicts a phoneme, the acoustic model faithfully renders the wrong sound. End-to-end systems aim to collapse some or all of these stages into a single model.
One approach converts audio into discrete tokens, similar to how text is broken into words or subwords. This transforms speech generation into something closer to a language modeling problem: predict the next audio token given the text and previous tokens. Models like VALL-E and Bark follow this approach, and it has simplified the TTS pipeline considerably.
There’s a tradeoff, though. Autoregressive models, which generate output one token at a time, tend to produce more natural-sounding speech but can hallucinate, skip words, or repeat phrases. Non-autoregressive models use an explicit duration predictor to avoid these problems but sometimes sound slightly less natural. Hybrid approaches, like VITS, try to get the best of both worlds by combining different generation strategies.
Voice Cloning and Customization
Recent systems can clone a voice from a short audio clip of someone speaking. These zero-shot voice cloning models extract the speaker’s unique vocal characteristics (timbre, cadence, pitch range) from just a few seconds of reference audio, then apply those characteristics when generating new speech. Both diffusion-based models and language-model-based approaches have achieved impressive results, capturing fine-grained speaker qualities from brief prompts.
This works because the models learn to separate “what is being said” from “who is saying it.” The speaker’s identity becomes a set of features that can be mixed and matched with any text content. Training typically uses audio clips up to about 30 seconds long, though inference can work with even shorter samples.
Measuring Quality
The standard way to evaluate synthetic speech is the Mean Opinion Score, where human listeners rate audio on a 1-to-5 scale across dimensions like overall quality, listening effort, articulation, pronunciation, speaking rate, and pleasantness. This framework was originally developed for evaluating telephone network audio quality and later adapted specifically for TTS evaluation.
Natural human speech typically scores around 4.5 on this scale (not a perfect 5, because listeners are tough graders). The best modern TTS systems score in the 4.0 to 4.5 range, meaning that in many contexts, listeners have difficulty distinguishing synthetic speech from recordings of real people. The gap has narrowed considerably in the last five years, driven almost entirely by improvements in neural acoustic models and vocoders.
Real-Time Performance
For conversational applications like voice assistants or AI phone agents, latency matters. Users expect a response within a few hundred milliseconds of finishing their sentence. The TTS system itself needs to begin producing audio quickly enough that the overall delay feels natural.
Most modern TTS systems can generate speech faster than real time on standard hardware, meaning they produce a second of audio in less than a second of computation. The key bottleneck has shifted from the vocoder (which used to be the slowest stage) to the acoustic model, particularly for long sentences where transformer attention computations scale with sequence length. Streaming approaches, where the system starts playing audio before the entire sentence is generated, help mask remaining latency in practice.