An explanation helps one user judge one output; observability shows a team how an AI feature behaves over time. What each should show. Neither explainable AI nor AI observability earns trust on its own, because they serve different people.
Neither explainable AI nor AI observability earns trust on its own, because they serve different people. An explanation helps one person judge one output while they are looking at it; observability tells the team accountable for an AI feature how it behaves across many outputs and from week to week. Give the person deciding something they can check, and give the team a weekly view of behaviour.
Checkable matters because a large language model's explanation may not be why it answered: in Anthropic's tests, reasoning models that used a planted hint mentioned it only 25% (Claude 3.7 Sonnet) and 39% (DeepSeek R1) of the time. A story about why is not evidence. The weekly view, for its part, works only when it is designed as a product surface rather than left in an engineering log.
What each one gives, and where to start
The NIST AI Risk Management Framework separates transparency ("what happened"), explainability ("how" a decision was made) and interpretability ("why", and what it means to the user). Observability, as used here, is the running record of what happened, kept over time and read by people who can act on it.
| Explanation | Observability | |
|---|---|---|
| Question it answers | Why did the system produce this output? | How is the system behaving across outputs, and is it changing? |
| Who acts on it | The person deciding about one output | The team accountable for the feature; sometimes a customer's administrator |
| When | At the moment of decision | Continuously, after launch |
| What it cannot do | Prove that the stated reason is the real one | Tell one user whether this answer is right |
| How it fails | A plausible reason makes a wrong output easier to accept | It counts speed, cost and approval, and misses a change in behaviour |
Where to invest first:
- If a person must stand behind each output (a claim they will repeat, a figure they will file, a record the system will change), start with what they can check at the moment of decision.
- If outputs flow at volume without anyone reading each one, or your model provider updates the model under you, start with observability for the team.
- If both are true, design them together. What users do with each check (open the source, edit the field, discard the answer) is itself a direct observability signal.
When an explanation misleads
Large language models produce explanations fluently. The trouble is that the explanation may not be why the model answered, and a convincing one makes people more willing to accept a wrong answer.
- Stated reasons can be unfaithful. Turpin and colleagues (NeurIPS 2023) biased models towards an answer, for example by reordering multiple-choice options so the answer was always "(A)". Of 426 explanations supporting biased answers that the authors reviewed, only one mentioned the bias. When biased towards wrong answers, the models (GPT-3.5 and Claude 1.0) frequently rationalised them, and accuracy fell by as much as 36% across 13 tasks.
- Reasoning models are no exception. The Anthropic figures above come from models that show their reasoning; the visible reasoning still left out the hint most of the time.
- Convincing explanations raise reliance on wrong answers. In a fact-checking study with 80 crowdworkers, Si and colleagues (NAACL 2024) found that people reading an LLM's explanation worked faster than people using a search engine, with similar accuracy, but over-relied on the model when its explanation was wrong. Explanations that argued both for and against the claim reduced over-reliance, without significantly beating plain search results.
- The guideline authors warn about it. Microsoft's guidance on its guideline to make clear why the system did what it did advises applying it judiciously, because the mere presence of an explanation has been shown to increase trust, which can lead to over-reliance.
Explanations are not useless. In five studies with 731 participants solving mazes with a simulated AI, Vasconcelos and colleagues (CSCW 2023) found situations where explanations do reduce over-reliance, and that the cost of engaging with an explanation shapes whether people rely on the AI. Their reading: an explanation helps when it makes the answer cheaper to verify.
That is the design move. Show the person deciding what they can check: the passage the answer came from, the input values it used, the one field that would be wrong if the whole output were wrong. In our AI Trust, Control and Recovery work we treat a rationale as an aid to checking, never a substitute for it, and where a system cannot establish why it produced a result, we leave the reason out rather than write a believable one.
Observability as a product surface: what to show each week
The model you depend on can change while your code does not, and approval can look healthy while it gets worse.
- Models shift under you. Chen, Zaharia and Zou (Harvard Data Science Review, 2024) compared the March and June 2023 versions of GPT-4: its accuracy at telling prime from composite numbers fell from 84% to 51%. They concluded that the "same" model service can change substantially in a short time and needs continuous monitoring.
- Approval can rise while behaviour gets worse. OpenAI's account of its April 2025 GPT-4o update describes an update, rolled out on 24 and 25 April, that made the model noticeably more sycophantic. Offline evaluations looked good and small A/B tests suggested users liked it; a new reward signal from thumbs-up and thumbs-down data was one of several changes that together may have tipped the balance. Rollback began on 28 April. OpenAI wrote that its A/B tests lacked the right signals, while hands-on checks had been hinting at the problem.
Samy Melaine, who founded Eden AI (a developer platform for many AI models, and our client) and is its CPTO, described the same shift on our podcast in an episode published in February 2025. Users first wanted explanations; as models grew, explaining an output from the model's internals kept getting harder; so his team moved to watching behaviour in production, and in his words "observability now is more crucial than explainability". They track how output changes from week to week, and for a chatbot they count how often the same question comes back, a sign the first answer failed, and compare the bot's vocabulary with the user's to spot rising frustration. His users are developers, so his conclusion is about the accountable team, not the person reading one answer.
Built from his method and the two cases above, this is what we would put in front of the people accountable for an AI feature:
| What to show each week | Why it matters | Who acts on it |
|---|---|---|
| What changed: model, prompt or retrieval version, and how output on a fixed sample of inputs differs from last week | Catches drift you did not cause | The engineering and product owners |
| Where users repeat themselves: the same question asked again, or rephrased | A repeated question is a failed answer that no error log records | The product owner |
| What users did with each output: accepted, edited, discarded or redone by hand | Separates use from reliance; approval alone can mislead | The product and design owners |
| The language of frustration in conversations | Shows how users feel before they leave | The product owner and support |
| Which model or route handled which request, if the system chooses | Lets someone answer "why did it do that?" across many outputs | Whoever answers to the customer |
| A small sample of real outputs to read | Metrics missed the GPT-4o problem; hands-on checks caught hints of it | Someone with the authority to stop a release |
What users do with each output is the subject of our article on why users stop using an AI feature after trying it once.
Where an explanation is still owed
The move to observability holds for the team. NIST's framework asks that an AI system's behaviour be monitored in production (MEASURE 2.4) and that post-deployment monitoring capture and evaluate input from users (MANAGE 4.1). A team that spends its effort on per-answer rationales and has no view of behaviour over time has the order wrong.
It does not hold in two places.
- The person deciding about one output. Observability arrives later and in aggregate; it cannot tell someone whether the figure in front of them is right. They still need something at the moment of decision, and the research above says it should be checkable rather than a narrative.
- Regulated decisions. Under Article 86 of the EU AI Act, a person affected by a decision a deployer takes on the basis of a high-risk system listed in Annex III (critical-infrastructure systems excepted), where the decision has legal or similarly significant effects they consider adverse to their health, safety or fundamental rights, can obtain "clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken". That is an explanation of the decision, not of the model's internals, and the same regulation requires providers of high-risk systems to run post-market monitoring (Article 72). In these uses a product needs both.
About the authors
Chirag Patel
Product Designer
More from ChiragChirag Patel is a Product Designer at Tcules who writes about AI product UX and the decisions behind it. Chirag's articles ask practical questions: should search and the AI assistant share one input, how should users choose a model, what changes when the interface itself is generated, and how should AI usage be billed and shown. Other pieces cover AI chatbot design for B2B products, admin consoles for governing AI agents, search in data-heavy SaaS, and whether designers should write prompts with ML engineers. Chirag has also co-written on keeping a front-end build true to its design.

Rushali Bhatt
Product Designer
More from RushaliRushali Bhatt is a Product Designer at Tcules. Rushali's articles start from the people inside complex software: clinicians and patients in healthcare products, users who cannot take part in research sessions, and the reviewers who check an AI feature's output before anyone acts on it. Topics include usability studies for enterprise software, human-in-the-loop patterns, a readiness checklist for AI product UX, what to fix first when adding AI to a live SaaS product, and what a heuristic evaluation should deliver. Useful reading for product teams whose software people rely on to make decisions.