Interpretability is the degree to which a human can understand why an AI system made a particular decision. In a world where algorithms help determine who gets a loan, how a medical image is read, and whether a traveler is flagged for screening, that understanding is not academic curiosity. A growing body of research argues that trying to explain opaque models after the fact is often unreliable, and that building systems whose reasoning is transparent by design should be the default for high-stakes decisions. Yet the field is far from settled on how to achieve that transparency, and the methods available each come with trade-offs that practitioners and policymakers need to appreciate.
Why Not Just Explain the Black Box?
The most direct argument in the interpretability debate is also the most provocative: stop explaining black-box models and use interpretable ones instead. That is the position laid out in a widely cited paper arguing that post-hoc explanation tools, which attempt to rationalize a complex model’s output after the fact, can perpetuate bad practices and risk catastrophic harm in domains like healthcare and criminal justice.1PubMed Central. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead The core concern is straightforward: if a model’s internal logic is hidden, any explanation layered on top is a guess about what the model “really” did, and that guess can be wrong in ways nobody notices until real people are harmed.
An inherently interpretable model, by contrast, is one whose decision logic you can inspect directly. Think of a scoring system a doctor can read on a card, or a decision tree short enough to trace by hand. The trade-off people assume is that these simpler models sacrifice accuracy. But a large benchmark study spanning twenty tabular datasets and tens of thousands of model runs found that interpretable models performed on par with black-box alternatives for tabular data, dispelling the notion that there is a strict trade-off between predictive performance and model interpretability.2Business & Information Systems Engineering. Challenging the Performance-Interpretability Trade-Off: An Evaluation of Interpretable Machine Learning Models A separate analysis reached a similar conclusion, finding that linear models with well-engineered features could match or beat complex ones on many benchmark tasks.3arXiv. Lifting Interpretability-Performance Trade-off via Automated Feature Engineering
This does not mean interpretable models win everywhere. For unstructured data like images, audio, and natural language, deep neural networks remain the dominant approach, and those networks are anything but transparent. The performance parity finding applies most clearly to structured, tabular data, which happens to cover a huge number of real-world decision-making scenarios: credit scoring, hospital readmission prediction, recidivism risk assessment, insurance pricing. In those domains, reaching for a black box when an interpretable model would do equally well is a choice, not a necessity.
When Transparency Builds Trust in a Crisis
Greece’s COVID-19 border-testing system offers a concrete case study. Researchers designed an interpretable operations-research model to allocate scarce rapid tests to arriving travelers, deliberately making every design and algorithmic choice emphasize transparent reasoning so that public health experts could follow the logic. That transparency was crucial to building trust and buy-in among policymakers during a period of global crisis.4SSRN Electronic Journal. Interpretable OR for High-Stakes Decisions: Designing the Greek COVID-19 Testing System If the system had been a black box recommending which travelers to test, officials would have had no way to verify it was not discriminating on the basis of nationality, or to adjust its priorities as the pandemic evolved. The interpretability was not a nice extra; it was what made the system deployable.
The Fragility of Post-Hoc Explanations
For the many settings where deep learning is genuinely needed, researchers have built a cottage industry of tools that try to explain black-box predictions after the fact. The two most widely used are LIME and SHAP, which work by perturbing inputs and observing how the model’s output changes. SHAP in particular has become a standard tool in fields like drug development, where practitioners use it to understand which input features drive a model’s predictions.5PubMed Central. Practical guide to SHAP analysis: Explaining supervised machine learning model predictions in drug development
The problem is that these perturbation-based explanations can be gamed. Researchers demonstrated a scaffolding technique that allows an adversary to wrap a biased classifier so that its predictions remain biased on real data, but the post-hoc explanations look completely innocuous. Using real-world datasets including the COMPAS recidivism tool, they showed that extremely biased, racist classifiers could fool LIME and SHAP into generating explanations that did not reflect the underlying biases at all.6arXiv. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods Later work confirmed the vulnerability and began developing defenses, though the arms race between attack and defense continues.7Proceedings of the AAAI Conference on Artificial Intelligence. Unfooling Perturbation-Based Post Hoc Explainers
The implication for anyone relying on SHAP or LIME in an audit or compliance context is sobering. An explanation that “looks right” is not evidence that the model is fair or correct. A malicious actor, or even an unintentionally misleading pipeline, can produce explanations that are plausible to human reviewers without reflecting the model’s actual reasoning.
Thinking in Concepts Instead of Features
Feature-attribution tools like SHAP tell you which input variables mattered, but for image classifiers or other models with thousands of raw inputs, a list of important pixels is not terribly useful to a person. Concept Activation Vectors (CAVs) take a different approach: they let you ask whether a human-friendly concept, such as “stripes” or “texture,” influenced a model’s prediction. The original TCAV method uses directional derivatives to quantify how sensitive a classification is to a given concept. For example, you could measure how much the concept “stripes” contributes to a model labeling an image as a zebra.8arXiv. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
Follow-up work introduced Robust Concept Activation Vectors (RCAV), which account for non-linearity in the model and apply stricter statistical testing. RCAV yields interpretations that are more accurate for individual images and more robust when evaluated across a full dataset.9arXiv. Robust Semantic Interpretability: Revisiting Concept Activation Vectors Concept-based methods feel more natural to domain experts: a radiologist cares whether the model is responding to “pleural effusion” as a concept, not whether pixel 47,812 had a high attribution score.
Mechanistic Interpretability and the Inner Life of Neural Networks
Post-hoc explanations and concept-based tools both treat the model as something to be probed from the outside. Mechanistic interpretability tries to go deeper: reverse-engineering the actual computations a neural network performs, the way an electrical engineer might trace a circuit. The ambition is to understand not just what a model predicts but how it arrives at that prediction internally.
One of the central puzzles is polysemanticity, the tendency of individual neurons to respond to multiple unrelated concepts. A single neuron might activate for both “cat photos” and “the word ‘because'” and “the color blue,” which makes it nearly impossible to assign a clean interpretation to any one neuron. Research using toy models has shown that polysemanticity arises because networks store more features than they have neurons, packing sparse features into a shared representational space through a phenomenon called superposition.10arXiv. Toy Models of Superposition Further analysis found that networks tend to represent the most important features cleanly (one neuron, one meaning) while polysemantically encoding less important features, and that the degree of polysemanticity depends on the sparsity and distribution of the inputs as well as the architecture.11arXiv. Polysemanticity and Capacity in Neural Networks
To untangle superposition, researchers have turned to sparse autoencoders (SAEs), which decompose a model’s internal activations into a larger set of interpretable features. Work on a one-layer transformer demonstrated that SAEs can extract a large number of individually interpretable features from a model’s internals.12Transformer Circuits Thread. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning Subsequent improvements, including gated sparse autoencoders, refined the technique further.13arXiv. Improving Dictionary Learning with Gated Sparse Autoencoders
Another landmark finding involves induction heads, attention-head circuits that implement a simple pattern-matching algorithm: if the model has seen the sequence A-B earlier in the text and now sees A again, the induction head copies B as the next prediction. This circuit has been identified as a key mechanism for in-context learning, the ability of language models to pick up new patterns from examples in a prompt.14arXiv. In-context Learning and Induction Heads Later work examined the specific conditions needed for induction heads to form and function correctly, treating them as a test case for understanding how interpretable circuits emerge during training.15arXiv. What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation
From Understanding to Steering
Mechanistic interpretability is not only about reading the circuitry; it can also be used to control model behavior. Activation steering modifies a model’s internal representations during inference to shift its outputs. One technique, Activation Addition, computes a “steering vector” by contrasting the model’s internal activations on paired prompts (like “Love” versus “Hate”) and then adds that vector during a forward pass to push the model’s output in a desired direction. This approach achieved strong results on sentiment shift and detoxification tasks across multiple large language models.16arXiv. Steering Language Models With Activation Engineering
More recent work combines steering with the sparse autoencoder features described above. Sparse activation steering operates in the decomposed feature space rather than raw activation space, which can give finer-grained control over behavior.17arXiv. Steering Large Language Model Activations in Sparse Spaces A related method derives detoxification vectors from self-induced steering pairs and fuses them with the model’s original representations in a head-by-head fashion during inference.18arXiv. DESTEIN: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion The practical upshot is that interpretability tools are evolving from passive observation into active intervention, letting developers adjust what a model does without retraining it from scratch.
When Explanations Meet Real People
An explanation is only useful if the person receiving it actually benefits from it, and the research here is more complicated than you might expect. One study examined how different types of AI explanations affected physicians reading chest X-rays. Local explanations, which highlight the specific image regions the AI focused on, led to better diagnostic accuracy when the AI’s advice was correct and reduced the time physicians spent evaluating AI suggestions. Physicians also placed greater trust in local explanations than in global ones, which describe the model’s general behavior. Strikingly, physicians were not consciously aware that the explanation type was influencing their performance and trust.19PubMed Central. Care to Explain? AI Explanation Types Differentially Impact Chest Radiograph Diagnostic Performance and Physician Trust in AI
But more explanation does not always mean better decisions. A study on automation bias found that providing AI explanations actually increased the rate at which people accepted incorrect AI recommendations. Commission errors, where a person follows bad AI advice they should have caught, were roughly twice as high in the explainable-AI setting (about 21%) as in the plain-AI setting (about 10%).20Thirtieth European Conference on Information Systems (ECIS 2022). On the Influence of Explainable AI on Automation Bias The explanation, rather than helping users think critically, seemed to give them a reason to trust the system more and question it less. This is a genuine design hazard: explanations that feel convincing can lull users into complacency.
Plausible Does Not Mean Faithful
A related concern applies to large language models that generate their own explanations in natural language. These models are skilled at producing explanations that sound logical and coherent to human readers, but research has found that such plausible-sounding explanations do not necessarily align with the model’s actual internal reasoning process.21arXiv. Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models In other words, the model may give you a perfectly articulate reason for its answer that has little to do with how it actually computed that answer. This gap between plausibility and faithfulness is one of the hardest problems in the field, because it means you cannot simply ask a model “why did you say that?” and trust the response.
Probing methods, where researchers train small classifiers on a model’s hidden states to detect what information is encoded there, have their own pitfalls. A recent study probing a large language model’s hidden states found that linear probes achieved perfect accuracy at distinguishing between different reasoning tasks. That looked like strong evidence that the model had learned distinct internal representations for different types of reasoning. But when the researchers controlled for surface-level format differences like the number of answer options and response length, probe accuracy dropped to chance. The geometry the probes had detected reflected task formatting, not computational structure.22arXiv. Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States The takeaway is that interpretability tools can sometimes tell you more about the experiment’s design than about the model’s mind.
Catching a Model in a Lie
One of the most consequential applications of interpretability is detecting when an AI system is being deceptive, either because it was deliberately trained to deceive or because safety fine-tuning created misalignment between its stated goals and its internal computations. Researchers have shown that linear probes trained on a language model’s internal activations can detect deceptive outputs with over 90% accuracy across models ranging from 1.5 billion to 14 billion parameters.23arXiv. Caught in the Act: a mechanistic approach to detecting deception
A complementary approach uses adversarial activation patching, which takes internal activations from prompts designed to elicit deceptive behavior and patches them into safe forward passes at specific layers. This lets researchers simulate and measure a model’s vulnerability to deception, then develop targeted interventions.24arXiv. Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformers As language models become more capable and are deployed in higher-stakes settings, the ability to look inside a model and verify that it is doing what it claims becomes a safety-critical capability, not just an intellectual exercise.
Regulation and the Right to an Explanation
The European Union’s AI Act, finalized in early 2024, mandates transparency and explainability requirements for AI systems, particularly those classified as high-risk. But translating legal requirements into technical practice is far from straightforward. A framework analyzing the Act’s provisions identified key friction points, including tensions between the need for meaningful explanations and the potential risks to intellectual property and commercial interests of AI providers.25Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. Habemus a Right to an Explanation: so What? – A Framework on Transparency-Explainability Functionality and Tensions in the EU AI Act A company might be legally required to explain how its model reaches a decision, but providing that explanation could reveal proprietary model architecture or training data. How regulators and courts resolve this tension will shape what “interpretability” means in practice for the next decade.
Drawing on How Humans Explain Things
The interpretability community has increasingly looked beyond computer science for guidance. Research drawing on philosophy, cognitive science, and the social sciences has surveyed how humans naturally explain things to each other and proposed design principles for machine-generated explanations based on those findings.26Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence Human explanations tend to be contrastive (“why this and not that?”), selective (highlighting one or two causes rather than listing every factor), and tailored to the listener’s knowledge. Most current interpretability tools do none of those things well. A SHAP plot dumps every feature’s contribution on the screen without any sense of what the human viewer already understands or what alternative outcome they are wondering about. Closing that gap between how machines currently explain and how humans naturally process explanations is arguably the most underappreciated frontier in the field.
Beyond Text and Tables
Much of the interpretability toolbox was developed for language models and tabular classifiers, but AI systems increasingly process images, video, and combinations of modalities simultaneously. A systematic review of mechanistic interpretability methods applied to vision-language models found that while many techniques transfer from the text-only setting, these multimodal systems present distinct challenges. The interaction between visual and linguistic representations adds layers of complexity that existing tools were not designed for, and the lack of scientific understanding of how these models work poses challenges for robustness and generalization, especially in high-stakes applications.27arXiv. Mechanistic Interpretability Meets Vision Language Models: Insights and Limitations As AI moves further into medical imaging, autonomous driving, and scientific discovery, the interpretability tools will need to keep pace with the models they are meant to explain.
Saliency Maps and the Problem of Validation
One of the oldest interpretability techniques for image models is the saliency map, a heatmap showing which pixels influenced a prediction. These maps are intuitively appealing but notoriously hard to validate. The Model Parameter Randomisation Test (MPRT) was designed as a basic sanity check: if you scramble a model’s parameters, the explanation should change, because the explanation should depend on what the model learned. But methodological concerns about how the test is implemented led researchers to propose improved versions, Smooth MPRT and Efficient MPRT, that reduce noise in the evaluation and avoid biased similarity measurements.28arXiv. A Fresh Look at Sanity Checks for Saliency Maps The fact that the community is still refining the basic tests for whether an explanation method works tells you something about how young the field remains. Interpretability is not just a set of tools. It is a set of tools whose reliability is itself an active area of research.