The biological species concept defines a species as a group of natural populations whose members can interbreed with one another but are reproductively isolated from members of other such groups. First formalized by the evolutionary biologist Ernst Mayr in the mid-twentieth century, it remains the most widely taught species definition in biology and the one most people encounter in textbooks. It is also, among working biologists, one of the most debated, because reproductive isolation turns out to be messier and more porous than the clean definition suggests.
What the Concept Actually Says
At its core, the biological species concept (BSC) makes reproductive isolation the defining feature of a species. Two populations belong to the same species if they can, at least in principle, mate and produce fertile offspring. They belong to different species if something prevents that gene exchange from happening under natural conditions. The emphasis on “natural conditions” matters: lions and tigers can produce offspring in captivity, but they do not interbreed in the wild, so the BSC treats them as separate species.
Mayr’s formulation was groundbreaking because it shifted the focus from what organisms look like to what they do reproductively. Earlier approaches classified species by physical appearance alone, which could lump together organisms that looked similar but never interbred, or split apart populations that looked different but freely exchanged genes. By anchoring the definition in gene flow, Mayr gave biologists a process-based criterion rooted in evolutionary theory rather than in the subjective judgment of a taxonomist sorting specimens in a museum drawer.
How Reproductive Isolation Works in Practice
Reproductive isolation is not a single barrier but a collection of mechanisms that can stack on top of one another. Biologists split them into two broad categories based on timing: barriers that prevent mating or fertilization from happening in the first place, and barriers that act after fertilization has already occurred.
Before Fertilization
Pre-zygotic barriers are the front line. They include differences in habitat preference, mating season, courtship behavior, or physical anatomy that keep populations from ever combining their genes. In marine organisms like brown seaweeds in the genus Fucus, closely related species share the same tidal cues for releasing gametes but shift the precise timing of that release within a single tide cycle, creating a narrow temporal window that limits hybridization.1PLoS ONE. Prezygotic Barriers to Hybridization in Marine Broadcast Spawners: Reproductive Timing and Mating System Variation In damselflies, physical differences in male clasping appendages can block copulation entirely; research on young damselfly lineages found that pre-mating barriers accounted for roughly 65 to 98 percent of total reproductive isolation, even before any post-mating incompatibilities kicked in.2Biological Journal of the Linnean Society. Rapid evolution of prezygotic barriers in non-territorial damselflies
These barriers do not always hold. Studies of parasitic trematodes found that two species hybridize freely under natural conditions, and in mate-choice experiments, individuals from both species actually preferred partners of the other species over their own.3PubMed Central. The role of prezygotic isolation mechanisms in the divergence of two parasite species Cases like this highlight that “reproductively isolated” is not always a binary state.
After Fertilization
Post-zygotic barriers come into play when mating does happen but the resulting offspring are inviable or infertile. Haldane’s rule, one of the oldest patterns in speciation biology, predicts that when one sex of a hybrid is sterile or inviable, it tends to be the sex with two different sex chromosomes. Genome mapping in Heliconius butterflies has shown that hybrid female sterility involves multiple interacting genetic regions, with the sex chromosome playing a large role alongside specific regions on other chromosomes.4PubMed. Complex basis of hybrid female sterility and Haldane’s rule in Heliconius butterflies: Z-linkage and epistasis In other words, hybrid breakdown is not the work of a single gene but of complex genetic clashes that build up as populations diverge.
Where the BSC Breaks Down
The biological species concept works well for many animals, particularly vertebrates and insects with clear sexual reproduction and stable geographic ranges. But life on Earth is far more varied than that, and several major groups of organisms expose the concept’s blind spots.
Organisms That Do Not Have Sex
The most obvious problem is that the BSC has nothing to say about organisms that reproduce asexually. If species are defined by interbreeding, what do you call a lineage that never interbreeds with anything? Brine shrimp in the genus Artemia include both sexual species and obligate parthenogenetic lineages that reproduce without males. These asexual forms cannot be classified under the BSC because concepts like “reproductive isolation” and “mate recognition” simply do not apply to them.5Zoological Journal of the Linnean Society. The species problem in Artemia Leach, 1819 (Crustacea: Anostraca), a genus with sexual species and obligate parthenogenetic lineages Some theorists have argued that sexual reproduction is so fundamental to the long-term stability of complex organisms that asexual lineages are inherently unstable and represent evolutionary dead ends, but that does not solve the practical problem of how to classify them while they exist.6PubMed Central. Biological species is the only possible form of existence for higher organisms: the evolutionary meaning of sexual reproduction
Microbes and Horizontal Gene Transfer
Bacteria and archaea present an even deeper challenge. These organisms routinely swap genetic material across lineage boundaries through horizontal gene transfer, a process fundamentally different from the vertical inheritance the BSC was designed around. Research on Campylobacter species has shown that the core genome of closely related bacterial species resists interspecies recombination, supporting the idea that something like species boundaries exist in microbes, but the non-core genome is far more fluid.7Genome Biology and Evolution. Evolutionary Dynamics of Complete Campylobacter Pan-Genomes and the Bacterial Species Concept The result is that closely related bacterial isolates can differ enormously in gene content even when they share most of their core genes, and the question of how cohesive populations evolve under such constant genetic shuffling remains genuinely controversial.8Trends in Genetics. Population genomics of early events in the ecological differentiation of bacteria
Fossils and Deep Time
You cannot test whether two populations interbreed when one of them has been extinct for fifty million years. This makes the BSC difficult to apply in paleontology. Some researchers have argued the fossil record is therefore silent on speciation, but others push back: fossilized organisms are sometimes preserved in large enough numbers that populations of gradually intergrading body forms can be recognized, and by analogy with living populations, these likely represent biological species.9Trends in Ecology & Evolution. Speciation in the fossil record Paleontologists working on Quaternary mammals have been urged to abandon the older practice of naming “chronospecies” based on time-sliced morphological change and instead align their classifications with the concepts used by biologists studying living organisms.10Quaternary International. The species concept and quaternary mammals
Hybridization and the Porous Species Boundary
Perhaps the most damaging challenge to a strict reading of the BSC is that hybridization between recognized species is far more common than Mayr’s framework assumed. Genomic data have made it clear that species boundaries are often semipermeable: some parts of the genome resist gene exchange, while others flow relatively freely between species.11Journal of Heredity. Hybridization, Introgression, and the Nature of Species Boundaries This is not just a curiosity. Introgression, the incorporation of genetic material from one species into the gene pool of another, can be adaptive. In plants, genomic tools have revealed that the movement of beneficial alleles between species may influence processes as wide-ranging as population rescue and adaptive radiation.12PubMed Central. Adaptive introgression: a plant perspective
What ecological, behavioral, and evolutionary factors determine which genetic variation moves between species, and whether that movement is biased in one direction, is an active area of research. The stakes are real: in a changing world, hybridization and introgression may determine which species persist and which go extinct.13Trends in Ecology & Evolution. Biased gene flow and the dynamic of introgression
Whole-genome duplication, or polyploidy, adds another wrinkle. In flowering plants, polyploidy is a major route to speciation and is often assumed to confer instant reproductive isolation because a newly polyploid individual has a different chromosome number from its parent population. But research on the plant genus Capsella found that polyploid formation did not actually create instant reproductive isolation, and shared genetic variation across ploidy levels could reflect ongoing gene exchange rather than simply multiple independent origins of the polyploid.14Molecular Biology and Evolution. Polyploid Speciation Did Not Confer Instant Reproductive Isolation in Capsella (Brassicaceae)
Ring Species and the Continuum Problem
Ring species have long been held up as textbook demonstrations of how one species can gradually evolve into two. In the greenish warbler complex, a ring of populations wraps around the Tibetan Plateau. At the ends of the ring, in central Siberia, two forms coexist without interbreeding. But along the southern arc connecting them, there is a gradient of genetic and physical characteristics with no clean break. Genomic analysis of this system has complicated the simple story, casting doubt on the idea that it represents speciation purely by geographic distance, while still confirming that the complex spans a continuum from slightly divergent neighboring populations to forms that are nearly fully reproductively isolated.15Nature. Genomic divergence in a ring species complex
Cases like this illustrate a deeper philosophical issue with the BSC: speciation is a process, not an event. At any given moment, some populations are midway through divergence, and asking whether they are “the same species” or “different species” may not have a clean answer. The BSC demands a yes-or-no verdict, but nature often offers a gradient.
Cryptic Species and Hidden Diversity
If the BSC sometimes lumps together populations that should be split, it also misses species that look identical but are reproductively isolated. DNA barcoding has exposed enormous hidden diversity. A study of deep-sea Antarctic polychaete worms found cryptic species in half of the morphological species examined, increasing the total species count in the sample by 233 percent.16PubMed Central. DNA barcoding uncovers cryptic diversity in 50% of deep-sea Antarctic polychaetes In tropical skipper butterflies, full nuclear genome sequencing confirmed that what had been treated as a single widespread species was actually three distinct species, each differing only subtly in appearance but showing significant genomic divergence and occupying different ecological niches.17PubMed Central. Nuclear genomes distinguish cryptic species suggested by their DNA barcodes and ecology Similar findings have emerged in reef fish, where DNA barcoding revealed three genetically divergent species hidden within what was thought to be a single species complex.18PubMed. Pseudogramma polyacantha complex (Serranidae, tribe Grammistini): DNA barcoding results lead to the discovery of three cryptic species, including two new species from French Polynesia
These discoveries do not disprove the BSC so much as show that its reliance on interbreeding needs molecular evidence to be applied properly. Populations that look alike under a microscope may have been evolving independently for thousands or millions of years. Without genetic tools, biologists applying the BSC would have no way to detect those hidden boundaries.
Alternative Species Concepts
Dissatisfaction with the BSC’s limitations has generated a long list of competing species definitions. Two of the most prominent are the phylogenetic species concept and the ecological species concept.
The phylogenetic species concept defines a species as the smallest diagnosable cluster of organisms within which there is a pattern of ancestry and descent. It does not require reproductive isolation; instead, it asks whether a group forms an exclusive evolutionary lineage. This approach handles asexual organisms, fossils, and microbes more gracefully than the BSC, but it can lead to aggressive splitting: minor geographic variants that the BSC would consider a single species may qualify as separate species under a strict phylogenetic reading.
The ecological species concept focuses on the niche a population occupies. If natural selection shapes two populations to exploit different resources, and that ecological divergence creates non-random mating, then the populations are on separate evolutionary trajectories. Research on Heliconius butterflies demonstrated that hybrid color patterns were attacked by predators more frequently than parental forms, meaning that ecological selection against intermediates directly reinforces the mating cue that keeps the species apart.19PubMed Central. Disruptive ecological selection on a mating cue This kind of coupling between ecology and mate choice is exactly the mechanism that ecological species concepts emphasize.
A growing number of biologists have argued that the way forward is not to pick one concept but to unify them. Under the general lineage concept, a species is simply a separately evolving segment of a population lineage. Reproductive isolation, diagnosability, monophyly, ecological distinctness: all of these are different lines of evidence that a lineage is evolving independently, not competing definitions of what a species is.20Systematic Biology. Species Concepts and Species Delimitation Under this framework, the various properties that different species concepts emphasize are treated as operational criteria rather than necessary conditions.21Endless Forms. The General Lineage Concept of Species, Species Criteria, and the Process of Speciation The more lines of evidence that converge, from genetics to ecology to reproductive behavior, the more confident you can be that you are looking at a genuine species. Mayr’s own definition, in this view, captures one important line of evidence but elevates it unnecessarily to the sole necessary condition.22PubMed Central. Ernst Mayr and the modern concept of species
Why the Definition Matters for Conservation
Species concepts are not just academic exercises. Legal frameworks for conservation, including the U.S. Endangered Species Act, are built around the concept of a species. When hybridization blurs the boundaries between listed and unlisted populations, the choice of species concept can determine whether a population receives legal protection. If strict adherence to the BSC treats hybrids as non-species, entire populations with conservation value could fall through the cracks.23Current Zoology. Hybridization and the species problem in conservation
Similarly, the discovery of cryptic species through DNA barcoding can reshape conservation priorities. If what was thought to be one widespread, low-risk species turns out to be three species with much smaller ranges, the threat level for each jumps. Genomic tools that detect “islands of divergence,” regions of the genome where differentiation is elevated between populations, are increasingly being used to refine the boundaries of management units for commercially exploited marine fish and other species.24PubMed Central. Genomic islands of divergence and their consequences for the resolution of spatial structure in an exploited marine fish
Less Obvious Drivers of Reproductive Isolation
Recent research has expanded the list of factors that can contribute to speciation beyond the classic genetic and ecological mechanisms. Epigenetic changes, chemical modifications that affect how genes are expressed without altering the DNA sequence itself, may contribute to reproductive isolation in some cases. These marks range in how faithfully they are passed from parent to offspring; some behave almost like genetic mutations, while others reset each generation. Where epigenetic marks are transmitted reliably, they could strengthen local isolation between populations even in the absence of underlying genetic differences, though their effects are expected to be weaker than those produced by genetic changes.25Journal of Evolutionary Biology. Epigenetics and reproductive isolation: a commentary on Westram et al., 2022
Intracellular symbionts can also play a role. The bacterium Wolbachia, which infects a vast proportion of insect species, can cause cytoplasmic incompatibility: matings between infected and uninfected individuals produce inviable embryos. In the apple maggot fly Rhagoletis, one Wolbachia strain is associated with a unique mitochondrial DNA type and causes one-directional post-mating incompatibility. Modeling suggests that this incompatibility alone would not prevent the strain from spreading between populations if they came into contact, but coupling between the symbiont and the host’s nuclear-mitochondrial interactions could impede gene flow under certain geographic scenarios.26PubMed Central. Testing the potential contribution of Wolbachia to speciation when cytoplasmic incompatibility becomes associated with host-related reproductive isolation In other words, a hitchhiking microbe inside a cell can nudge two populations toward reproductive isolation, a mechanism Mayr never anticipated.
Environment can even reshape mate preferences in hybrids themselves. In spadefoot toads, hybrid females preferred the calls of sterile hybrid males over those of one parent species when tested in deep-water conditions, a preference that could reinforce isolation by wasting the reproductive effort of hybrids. But in shallow water, that preference vanished.27PubMed Central. Hybrid female mate choice as a species isolating mechanism: environment matters The finding is a reminder that reproductive isolation is not just a property of genomes; it depends on the ecological stage where the drama plays out.
Allopatric Speciation and Why Geography Still Dominates
Despite the intellectual excitement around sympatric speciation, where new species arise within the same geographic area, the most common mode of speciation by a wide margin involves geographic separation. A river changes course, a glacier divides a forest, a few individuals colonize an island: the resulting isolation allows populations to diverge genetically and behaviorally until, if they ever come back into contact, they can no longer interbreed. Surveys of sister species across many groups have consistently found that they are rarely sympatric, implying that speciation in the absence of geographic barriers is genuinely uncommon, even though population-genetic models show it is theoretically possible.28PubMed. Pattern, process and geographic modes of speciation
This observation actually lends strength to the BSC in one respect: most species do pass through a phase where physical separation prevents gene flow, and by the time they come back into contact, reproductive isolation has often accumulated to the point where interbreeding is rare or absent. The concept works best for exactly these cases. Its struggles begin at the margins, with populations caught in the act of diverging, with taxa that defy the assumption of sexual reproduction, and with the growing genomic evidence that no species boundary is entirely leak-proof.