Observability or explainability: which earns trust in an AI product?

An explanation helps one user judge one output; observability shows a team how an AI feature behaves over time. What each should show.

  • AI Product UX
  • AI products and developer tools
  • AI Products & Developer Tools
Insights

An explanation helps one user judge one output; observability shows a team how an AI feature behaves over time. What each should show. Neither explainable AI nor AI observability earns trust on its own, because they serve different people.

Neither explainable AI nor AI observability earns trust on its own, because they serve different people. An explanation helps one person judge one output while they are looking at it; observability tells the team accountable for an AI feature how it behaves across many outputs and from week to week. Give the person deciding something they can check, and give the team a weekly view of behaviour.

Checkable matters because a large language model's explanation may not be why it answered: in Anthropic's tests, reasoning models that used a planted hint mentioned it only 25% (Claude 3.7 Sonnet) and 39% (DeepSeek R1) of the time. A story about why is not evidence. The weekly view, for its part, works only when it is designed as a product surface rather than left in an engineering log.

What each one gives, and where to start

The NIST AI Risk Management Framework separates transparency ("what happened"), explainability ("how" a decision was made) and interpretability ("why", and what it means to the user). Observability, as used here, is the running record of what happened, kept over time and read by people who can act on it.

ExplanationObservability
Question it answersWhy did the system produce this output?How is the system behaving across outputs, and is it changing?
Who acts on itThe person deciding about one outputThe team accountable for the feature; sometimes a customer's administrator
WhenAt the moment of decisionContinuously, after launch
What it cannot doProve that the stated reason is the real oneTell one user whether this answer is right
How it failsA plausible reason makes a wrong output easier to acceptIt counts speed, cost and approval, and misses a change in behaviour

Where to invest first:

  1. If a person must stand behind each output (a claim they will repeat, a figure they will file, a record the system will change), start with what they can check at the moment of decision.
  2. If outputs flow at volume without anyone reading each one, or your model provider updates the model under you, start with observability for the team.
  3. If both are true, design them together. What users do with each check (open the source, edit the field, discard the answer) is itself a direct observability signal.

When an explanation misleads

Large language models produce explanations fluently. The trouble is that the explanation may not be why the model answered, and a convincing one makes people more willing to accept a wrong answer.

  • Stated reasons can be unfaithful. Turpin and colleagues (NeurIPS 2023) biased models towards an answer, for example by reordering multiple-choice options so the answer was always "(A)". Of 426 explanations supporting biased answers that the authors reviewed, only one mentioned the bias. When biased towards wrong answers, the models (GPT-3.5 and Claude 1.0) frequently rationalised them, and accuracy fell by as much as 36% across 13 tasks.
  • Reasoning models are no exception. The Anthropic figures above come from models that show their reasoning; the visible reasoning still left out the hint most of the time.
  • Convincing explanations raise reliance on wrong answers. In a fact-checking study with 80 crowdworkers, Si and colleagues (NAACL 2024) found that people reading an LLM's explanation worked faster than people using a search engine, with similar accuracy, but over-relied on the model when its explanation was wrong. Explanations that argued both for and against the claim reduced over-reliance, without significantly beating plain search results.
  • The guideline authors warn about it. Microsoft's guidance on its guideline to make clear why the system did what it did advises applying it judiciously, because the mere presence of an explanation has been shown to increase trust, which can lead to over-reliance.

Explanations are not useless. In five studies with 731 participants solving mazes with a simulated AI, Vasconcelos and colleagues (CSCW 2023) found situations where explanations do reduce over-reliance, and that the cost of engaging with an explanation shapes whether people rely on the AI. Their reading: an explanation helps when it makes the answer cheaper to verify.

That is the design move. Show the person deciding what they can check: the passage the answer came from, the input values it used, the one field that would be wrong if the whole output were wrong. In our AI Trust, Control and Recovery work we treat a rationale as an aid to checking, never a substitute for it, and where a system cannot establish why it produced a result, we leave the reason out rather than write a believable one.


Observability as a product surface: what to show each week

The model you depend on can change while your code does not, and approval can look healthy while it gets worse.

  • Models shift under you. Chen, Zaharia and Zou (Harvard Data Science Review, 2024) compared the March and June 2023 versions of GPT-4: its accuracy at telling prime from composite numbers fell from 84% to 51%. They concluded that the "same" model service can change substantially in a short time and needs continuous monitoring.
  • Approval can rise while behaviour gets worse. OpenAI's account of its April 2025 GPT-4o update describes an update, rolled out on 24 and 25 April, that made the model noticeably more sycophantic. Offline evaluations looked good and small A/B tests suggested users liked it; a new reward signal from thumbs-up and thumbs-down data was one of several changes that together may have tipped the balance. Rollback began on 28 April. OpenAI wrote that its A/B tests lacked the right signals, while hands-on checks had been hinting at the problem.

Samy Melaine, who founded Eden AI (a developer platform for many AI models, and our client) and is its CPTO, described the same shift on our podcast in an episode published in February 2025. Users first wanted explanations; as models grew, explaining an output from the model's internals kept getting harder; so his team moved to watching behaviour in production, and in his words "observability now is more crucial than explainability". They track how output changes from week to week, and for a chatbot they count how often the same question comes back, a sign the first answer failed, and compare the bot's vocabulary with the user's to spot rising frustration. His users are developers, so his conclusion is about the accountable team, not the person reading one answer.

Built from his method and the two cases above, this is what we would put in front of the people accountable for an AI feature:

What to show each weekWhy it mattersWho acts on it
What changed: model, prompt or retrieval version, and how output on a fixed sample of inputs differs from last weekCatches drift you did not causeThe engineering and product owners
Where users repeat themselves: the same question asked again, or rephrasedA repeated question is a failed answer that no error log recordsThe product owner
What users did with each output: accepted, edited, discarded or redone by handSeparates use from reliance; approval alone can misleadThe product and design owners
The language of frustration in conversationsShows how users feel before they leaveThe product owner and support
Which model or route handled which request, if the system choosesLets someone answer "why did it do that?" across many outputsWhoever answers to the customer
A small sample of real outputs to readMetrics missed the GPT-4o problem; hands-on checks caught hints of itSomeone with the authority to stop a release

What users do with each output is the subject of our article on why users stop using an AI feature after trying it once.


Where an explanation is still owed

The move to observability holds for the team. NIST's framework asks that an AI system's behaviour be monitored in production (MEASURE 2.4) and that post-deployment monitoring capture and evaluate input from users (MANAGE 4.1). A team that spends its effort on per-answer rationales and has no view of behaviour over time has the order wrong.

It does not hold in two places.

  • The person deciding about one output. Observability arrives later and in aggregate; it cannot tell someone whether the figure in front of them is right. They still need something at the moment of decision, and the research above says it should be checkable rather than a narrative.
  • Regulated decisions. Under Article 86 of the EU AI Act, a person affected by a decision a deployer takes on the basis of a high-risk system listed in Annex III (critical-infrastructure systems excepted), where the decision has legal or similarly significant effects they consider adverse to their health, safety or fundamental rights, can obtain "clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken". That is an explanation of the decision, not of the model's internals, and the same regulation requires providers of high-risk systems to run post-market monitoring (Article 72). In these uses a product needs both.

About the authors

  • Chirag Patel

    Product Designer

    Chirag Patel is a Product Designer at Tcules who writes about AI product UX and the decisions behind it. Chirag's articles ask practical questions: should search and the AI assistant share one input, how should users choose a model, what changes when the interface itself is generated, and how should AI usage be billed and shown. Other pieces cover AI chatbot design for B2B products, admin consoles for governing AI agents, search in data-heavy SaaS, and whether designers should write prompts with ML engineers. Chirag has also co-written on keeping a front-end build true to its design.

    More from Chirag
  • Rushali Bhatt

    Product Designer

    Rushali Bhatt is a Product Designer at Tcules. Rushali's articles start from the people inside complex software: clinicians and patients in healthcare products, users who cannot take part in research sessions, and the reviewers who check an AI feature's output before anyone acts on it. Topics include usability studies for enterprise software, human-in-the-loop patterns, a readiness checklist for AI product UX, what to fix first when adding AI to a live SaaS product, and what a heuristic evaluation should deliver. Useful reading for product teams whose software people rely on to make decisions.

    More from Rushali

See what your users can check and what your team can see

Introduce us to the AI feature your users are asked to rely on.