AI decision makers already influence whether you get a loan, what medical diagnosis your doctor considers first, and how fast a stock trade executes. These systems range from simple rule-based filters to large language models that weigh millions of variables at once, and their track record is a genuine mix of impressive speed and troubling blind spots. When tested on reasoning tasks involving common cognitive traps, even top-performing large language models produced correct, well-reasoned answers only about half the time, suggesting that “artificial intelligence” and “reliable decision-making” are not yet synonyms.
How AI Decisions Differ From Human Ones
A human deciding whether to approve a mortgage application draws on training, gut feeling, institutional memory, and a rough sense of the applicant sitting across the desk. An AI decision maker draws on patterns extracted from historical data. It has no gut feeling and no contextual awareness of the room. What it does have is the ability to process thousands of variables simultaneously, apply them consistently across every single case, and never get tired at 4 p.m. on a Friday. That consistency is the main selling point, but it is also the source of most problems.
Researchers comparing AI agents to humans in a collaborative commuting game found that large language models could learn from past rounds and converge on good individual choices in ways that looked strikingly human-like. But the models struggled with group dynamics: they had weak awareness of what other players were choosing, poor collaborative reasoning, and little understanding of physical constraints that a person would take for granted.1PubMed Central. Comparing AI and human decision-making mechanisms in daily collaborative experiments In other words, AI can optimize for a well-defined individual goal, but it tends to fall apart where decision-making requires reading a social situation or coordinating with other agents whose behavior is unpredictable.
When researchers subjected several leading large language models to classic reasoning problems designed to trigger cognitive biases, GPT-4o and Gemma 2 each answered correctly about 49% of the time, while Llama 3.1 managed roughly 34%.2arXiv. Heuristics and Biases in AI Decision-Making: Implications for Responsible AGI Those numbers are not catastrophic, but they make clear that AI models carry their own version of the mental shortcuts that trip up human thinkers. The biases are different in flavor, often reflecting statistical quirks in training data rather than emotional impulses, but they are biases all the same.
The Bias AI Inherits From Us
An AI decision maker learns from data, and data is a fossil record of human behavior, including discriminatory behavior. If a bank’s historical lending records show that certain demographic groups were denied loans at higher rates, a machine learning model trained on those records will learn that pattern and replicate it. Researchers analyzing credit scoring models demonstrated that social biases embedded in application data were reliably transmitted into the model’s predictions: the algorithm could predict a customer’s gender or ethnicity from the same information used to assess creditworthiness.3AI and Ethics. Societal bias reinforcement through machine learning: a credit scoring perspective The model was not “trying” to discriminate. It was doing exactly what it was built to do: find patterns in historical data and extend them into the future.
This is the central tension in algorithmic fairness. A well-performing model, by standard accuracy metrics, can also be a deeply unfair one. Reviews of fairness research have emphasized that algorithmic decision-making can be inherently prone to unfairness even when no one involved intends it, because the data itself encodes decades of structural inequality.4ACM Computing Surveys. A Review on Fairness in Machine Learning Fixing bias after the fact is possible but tricky: you can adjust thresholds, re-weight training examples, or audit outputs for disparate impact, but each fix involves trade-offs with accuracy or with other definitions of fairness that may conflict with the one you chose.
Two Opposite Problems When Humans and AI Share the Wheel
When people work alongside AI decision makers, two psychological tendencies pull in opposite directions. The first is automation bias, the habit of deferring too readily to whatever the machine recommends. As AI has moved into high-stakes fields like healthcare, law, and public administration, automation bias has become a serious concern: people treat the AI’s output as a default answer rather than one input among several.5AI & SOCIETY. Exploring automation bias in human–AI collaboration: a review and implications for explainable AI The risk is obvious. If the AI is wrong and the human rubber-stamps it, no one in the loop actually made a decision.
The second tendency is algorithm aversion, a reflexive distrust of algorithmic recommendations. A systematic review of algorithm aversion identified five recurring themes driving it: mismatched expectations about how the algorithm should perform, a desire to retain personal control over the decision, differences in how humans and machines weigh reasons, and a basic sense that delegating a choice to a machine feels wrong.6Journal of Behavioral Decision Making. A systematic review of algorithm aversion in augmented decision making Interestingly, people who have seen an algorithm make even a single mistake often lose confidence in it far more sharply than they would after seeing a human make the same mistake. We hold machines to a standard of perfection that we would never apply to a colleague.
Both tendencies undermine good outcomes. Over-reliance means the AI’s errors go unchecked. Under-reliance means the AI’s strengths go unused. The sweet spot, where a human genuinely evaluates the AI’s recommendation before acting, is surprisingly hard to engineer. One approach that shows promise involves giving the human decision maker a shared picture of what the AI “sees” and why it reached its conclusion. Research on AI recommender systems found that when users were given tools to build shared situation awareness with their AI partner, their own task performance improved, their trust became better calibrated, and they were less likely to blindly follow a wrong recommendation.7Cognition, Technology & Work. Exploring the role of judgement and shared situation awareness when working with AI recommender systems
AI in the Exam Room
Healthcare is one of the highest-stakes arenas for AI decision-making, and the evidence from clinical studies is both encouraging and alarming, sometimes within the same experiment. A study examining how AI diagnostic recommendations influenced clinicians found that the AI’s effect was powerful in both directions: participants were roughly ten times more likely to reach the correct diagnosis when the AI recommendation was correct, but their accuracy dropped sharply when the AI pointed them in the wrong direction.8International Journal of Medical Informatics. Impact of AI recommendation correctness on diagnostic accuracy in clinical decision-making That tenfold boost sounds wonderful until you consider the flip side: a wrong AI suggestion can actively make a trained clinician worse at their job, at least in the moment. The system amplifies whatever signal it sends, correct or not.
This connects directly to the automation bias problem. In a diagnostic setting, a physician who trusts the AI’s recommendation may anchor on it so strongly that they skip the mental work of generating alternative diagnoses. When the AI happens to be right, this shortcut saves time and may catch things a tired human would miss. When the AI is wrong, the physician has effectively outsourced their reasoning to a system that cannot explain its own uncertainty. Disaster-scenario research has shown that well-designed AI support systems can cut cognitive load and speed up decision-making under pressure, but those benefits depend entirely on the human retaining enough independent judgment to push back when the AI’s suggestion does not match the situation on the ground.9arXiv. Human-AI Use Patterns for Decision-Making in Disaster Scenarios: A Systematic Review
The Deskilling Risk
One of the less discussed consequences of routine AI-assisted decision-making is what happens to the human’s own skills over time. If a doctor, a financial analyst, or an air-traffic controller consistently relies on an AI system to generate the first draft of a judgment call, the cognitive muscles that used to do that work can atrophy. This is not speculation. A scoping review examining the risk of deskilling among physicians found direct evidence of behavioral dependence after repeated AI use. In one notable example from endoscopy, clinicians’ ability to detect adenomas dropped from about 28% before AI was introduced to roughly 22% when they subsequently worked without AI support, even though their detection rate had been around 25% while using the AI tool.10PubMed Central. Artificial intelligence in medicine: a scoping review of the risk of deskilling and loss of expertise among physicians In other words, after a period of AI-assisted work, the doctors performed worse on their own than they had before the AI arrived.
A broader mixed-method review of AI-induced deskilling in medicine described both the erosion of existing expertise and a subtler problem: upskilling inhibition, where junior clinicians never fully develop certain diagnostic skills because the AI handles those tasks from the start of their training.11Artificial Intelligence Review. AI-induced Deskilling in Medicine: A Mixed-Method Review and Research Agenda for Healthcare and Beyond The concern extends well beyond medicine. Any field where practitioners develop expertise through repeated judgment calls, from underwriting to piloting to legal analysis, faces a version of the same risk when AI takes over the routine decisions that once served as training ground.
Why “Explain Yourself” Is Harder Than It Sounds
A recurring demand in AI governance is that automated decision makers should be explainable. If an AI denies your loan or flags your scan as abnormal, you should be able to ask why. In practice, delivering on that demand is genuinely difficult, and the field of explainable AI is more fragmented than it appears from the outside.
The core trade-off is between performance and interpretability. Simpler models like decision trees or logistic regressions are easy to understand but often less accurate on complex tasks. More powerful models, particularly deep neural networks, can capture subtle patterns that simpler models miss, but their internal reasoning is essentially opaque. To bridge the gap, researchers have developed “post-hoc” explanation tools that attempt to interpret a black-box model’s output after the fact. The problem is that different post-hoc methods, applied to the same model on the same data, can produce systematically different explanations. A viewpoint paper in a leading AI journal described this disagreement as a fundamental limitation and argued for a shift toward neural network architectures that are intrinsically interpretable, rather than building complex models and then bolting on explanations afterward.12Journal of Artificial Intelligence Research. Explainable Artificial Intelligence: A Call for Action in Human-Centric Fields
Guidelines for data scientists choosing between explanation approaches have noted that the decision often comes down to what matters most. If raw predictive accuracy is the priority, larger and more complex models tend to win, but they then require post-hoc explanation tools whose outputs may not fully agree with each other. If simplicity and understandability are also important, smaller “ante-hoc” models that are transparent by design may be the better choice, even at some cost in accuracy.13Cognitive Systems Research. Post-hoc vs ante-hoc explanations: xAI design guidelines for data scientists For a person on the receiving end of an AI decision, the practical takeaway is that “the AI explained its reasoning” does not guarantee the explanation is complete or that a different explanation tool would have told the same story.
Flash Crashes and Financial Markets
Financial trading is the domain where AI decision-making operates at its most extreme speed. Algorithmic trading systems execute thousands of transactions per second, reacting to price signals faster than any human could perceive. The upside is liquidity and efficiency. The downside is that when something goes wrong, it goes wrong very fast.
The most famous example remains the 2010 Flash Crash, in which the U.S. stock market plunged nearly a thousand points in minutes before bouncing back. Analysis of the event found that it exhibited the hallmarks of a “normal accident,” a term borrowed from systems theory for failures that arise not from any single broken component but from complex, tightly coupled interactions among many participants. Safeguards that were supposed to prevent cascading failures, including pre-programmed trading pauses and exchange circuit breakers, actually created new interactions that made things worse. Similar flash events have recurred since, including a 2014 crash in U.S. Treasury securities that saw prices spike and collapse within twelve minutes for no apparent reason.14PubMed Central. Systemic failures and organizational risk management in algorithmic trading: Normal accidents and high reliability in financial markets The research consensus is that these events are not freak occurrences but a predictable feature of markets where tightly coupled AI agents interact at superhuman speed.
What makes financial AI decision-making particularly interesting is that the individual algorithms are often working exactly as designed. Each one is optimizing for its owner’s defined objective: buy when the signal says buy, sell when the signal says sell, and do it faster than the competition. The systemic risk emerges from the interaction of many individually rational agents, not from any one agent malfunctioning. This is a pattern worth watching in other domains as AI decision makers multiply.
Who Is Liable When an AI Decides Wrong
Traditional legal frameworks assign responsibility to people. Someone designed the product, someone sold it, someone used it. When an AI system causes harm, figuring out who bears responsibility is genuinely complicated because the system’s behavior may not have been predictable by any single party in the chain. Legal scholars have identified three distinct gaps in existing liability rules for AI. First, it can be unclear what counts as a “defect” when an AI’s harmful output was not reasonably foreseeable by its designers. Second, proving causation is difficult when there is no traceable, predictable line between how the AI was designed and the specific harm it produced. Third, the types of damage AI can cause may not fit neatly into recognized legal categories of recoverable harm.15Computer Law & Security Review. The law and economics of AI liability
The problem deepens when you consider the supply chain behind a modern AI system. One company supplies the training data, another builds the model architecture, a third fine-tunes it for a specific application, and a fourth deploys it in a consumer product. Under traditional product liability, a component supplier is generally not liable for harm caused by the finished product. But AI systems can develop emergent properties, behaviors that arise from the interaction of components that none of the individual suppliers anticipated. This makes it extremely hard to assign responsibility among data providers, model developers, and system integrators.16Digital Law Journal. Artificial Intelligence and Liability: Exploring Legal Challenges and Responsibility in AI Decision-Making and Autonomous Systems Legal researchers have proposed several frameworks to address these gaps, including shared liability models where responsibility is distributed across the supply chain, risk-based liability that ties responsibility to the level of autonomy the AI exercises, and stronger algorithmic accountability requirements that force developers to document and audit decision processes.17Formosa Journal of Applied Sciences. Human Machine Liability Redistribution in Autonomous Artificial Intelligence Decision Systems within Contemporary Legal Governance
None of these proposals has become the dominant legal standard yet. In practice, liability for AI decisions currently depends heavily on jurisdiction and on whether the AI is classified as a product, a service, or something else entirely. For anyone building or deploying AI decision systems today, the legal landscape is one of the biggest unresolved risks.
Embedding Ethics Before Deployment
Most conversations about AI ethics happen after something goes wrong: a biased hiring tool makes the news, a self-driving car is involved in a fatal crash, a content recommendation engine radicalizes users. A growing body of work argues that ethical considerations need to be built into the development process itself, not patched on afterward.
One concrete approach is the ECCOLA method, a card-based framework designed to be used by software development teams during their normal workflow. Each card prompts the team to consider a specific ethical dimension of the system they are building. Evaluations of ECCOLA found that it worked in two main ways: it raised the team’s awareness of ethical issues they might not have considered, and it produced documentation of the ethical reasoning behind design choices, creating a paper trail that would otherwise not exist.18Journal of Systems and Software. ECCOLA — A method for implementing ethically aligned AI systems That documentation matters because one of the persistent problems with AI decision systems is that no one can reconstruct, after the fact, why certain trade-offs were made during development.
The broader challenge is that “ethical AI” is not a single technical property you can optimize for, the way you optimize for accuracy or speed. Different ethical values, such as fairness, transparency, privacy, and autonomy, can conflict with each other and with performance goals. A model that is maximally transparent may be less accurate. A model that protects individual privacy may be less fair at the group level because it cannot detect and correct for demographic disparities it is not allowed to see. These trade-offs do not have universal right answers, which is precisely why they need deliberate human judgment rather than default engineering choices made under deadline pressure.
When Multiple AI Agents Interact
Most public discussion of AI decision-making imagines a single system making a single choice. Increasingly, the real-world picture involves multiple AI agents operating in the same environment, each pursuing its own objective, and their interactions generating outcomes that no individual agent was designed to produce. Multi-agent reinforcement learning research has focused on understanding how groups of AI agents can learn to make good collective choices in complex environments, blending techniques from game theory, optimization, and biological models of swarm behavior.19PubMed Central. Multi-Agent Reinforcement Learning in Games: Research and Applications
The flash crash example from financial markets is one illustration of what happens when multi-agent AI coordination fails. But the same dynamic applies to autonomous vehicles sharing a road, to competing recommendation algorithms shaping a social media feed, or to supply chain management systems from rival companies interacting through the same logistics network. In each case, the behavior of the whole is not simply the sum of the individual agents’ behaviors. Emergent dynamics, feedback loops, and unintended coordination (or catastrophic failure to coordinate) are features of the landscape that single-agent thinking about AI misses entirely. As AI decision makers multiply across industries, the interactions between them may matter more than the capabilities of any one system in isolation.