Observability or explainability: which earns trust in an AI product?

Neither does it alone, because they serve different people at different moments. An explanation helps one person decide about one output while they are looking at it; observability tells the people accountable for an AI feature how it behaves across many outputs and over weeks. With large language models a fluent explanation can be unfaithful to what actually produced the answer, so for the person deciding, show what they can check rather than a story about why; for the team, design monitoring as a product surface rather than leaving it in an engineering log.

What each one gives, and to whom

The NIST AI Risk Management Framework (January 2023) separates three ideas: transparency answers "what happened", explainability answers "how" a decision was made, and interpretability answers "why" and what it means to the user. Observability, in the sense used here, is the running record of what happened, kept over time.

The rest of this page takes each column in turn, starts from an AI platform founder's account, and ends with a rule for where to invest first.

Why an AI platform moved from explaining to watching

At Tcules, we host a podcast, Beyond the Interface. In its first episode I asked Samy Melaine, founder and CPTO of Eden AI, which he described then as one developer platform for many AI models, how his team earns users' trust in AI output. Eden AI is our client, as I said when introducing him; our case study covers the search we designed for its workflow builder. The conversation was recorded around November 2024 and published on 26 February 2025, so what follows is his view at that date.

He described observability and monitoring as the third layer of their product, after choosing models and chaining them into workflows. His account of the shift ran like this:

• At first, people wanted explanations. Users and business people wanted to understand what the AI was doing and why.
• Explanation got harder as models grew. With large language models, he said, understanding why a model produced something by looking inside it became harder and harder.
• So the focus moved to behaviour in production. That meant seeing how the model behaves once it is live: for a text model, whether its output is biased, and from week to week, whether it shifted. In his words, "observability now is more crucial than explainability".

The method is the more useful part. Rather than build their own tools, they were working out how to aggregate partners' methods, because many startups measure this and each calculates it differently. They watch how output changes week to week. For a chatbot, they count how often the same question is asked, a sign the bot did not give the expected answer, and compare the bot's vocabulary with the user's to see whether the user is getting angry. Where the platform chooses the model itself, he said they show in the monitoring which models were chosen, with as much as they can about why.

Two things about this account matter for the argument. His users are developers building AI features into their own products, so his "users" are the team accountable for the feature. And his reason for the shift is about the model's internals, not about the person reading an output. Both limit how far his conclusion travels, which the research below makes clearer.

When an explanation misleads

The case against leaning on explanations is not that they are hard to produce. Large language models produce them fluently. The case is that the explanation a model gives may not be the reason it gave the answer.

• Stated reasons can be unfaithful. Turpin and colleagues (NeurIPS 2023) biased models towards an answer, for example by reordering multiple-choice options so the answer was always "(A)". Of 426 explanations supporting biased answers that the authors reviewed, only one mentioned the bias. When biased towards wrong answers, the models (GPT-3.5 and Claude 1.0) frequently gave explanations rationalising them, and accuracy fell by as much as 36% on a suite of 13 tasks.
• Reasoning models are not exempt. In Anthropic's tests (April 2025), models given a hint that they then used mentioned it in their visible reasoning only 25% of the time on average for Claude 3.7 Sonnet and 39% for DeepSeek R1.
• Convincing explanations raise reliance on wrong answers. In a fact-checking study with 80 crowdworkers, Si and colleagues (NAACL 2024) found that people reading an LLM's explanation worked more efficiently than people using a search engine, with similar accuracy, but people over-relied on the model when its explanation was wrong. Explanations that argued both for and against the claim reduced over-reliance, but did not significantly outperform plain search results.
• The guideline authors say so too. Microsoft's own guidance on its guideline 11, make clear why the system did what it did, advises applying it judiciously because the mere presence of an explanation has been shown to increase trust, which can lead to over-reliance.

None of this means explanations are useless. Vasconcelos and colleagues (CSCW 2023), in five studies with 731 participants solving mazes with a simulated AI, showed that there are situations where explanations do reduce over-reliance, and that how costly an explanation is to engage with affects whether people rely on the AI. They suggest that some earlier null results may be due in part to explanations that did not make the AI's answer cheaper to verify.

That points to the design move. For the person deciding about one output, the useful thing to show is what they can check: the passage the answer came from, the input values it used, the one field that would be wrong if the whole output were wrong. At Tcules, we hold the same position in our AI Trust, Control and Recovery work: explanation can help, but it is not verification, and it can make an output easier to accept without making it easier to check. Where a system cannot establish why it produced a result, we do not design a plausible account of it.

Observability as a product surface, not an engineering log

Observability matters for a reason beyond the size of the model: the model you depend on can change while your code does not.

• Models shift under you. Chen, Zaharia and Zou (Harvard Data Science Review, 2024) compared March and June 2023 versions of GPT-4 and GPT-3.5. GPT-4's accuracy at telling prime from composite numbers fell from 84% to 51%, and the authors concluded that the behaviour of the "same" model service can change substantially in a short time, so it needs continuous monitoring.
• Approval can rise while behaviour gets worse. OpenAI's account of its April 2025 GPT-4o update describes an update, rolled out on 24 and 25 April, that made the model noticeably more sycophantic. Offline evaluations generally looked good and small A/B tests suggested users liked it; an additional reward signal built from thumbs-up and thumbs-down data was one of several changes that, combined, may have tipped the balance. They began rolling it back on 28 April, and wrote that their A/B tests lacked the right signals and that the qualitative checks had been hinting at the problem.

Standard telemetry does not see this kind of change by itself. The OpenTelemetry generative AI metrics conventions, still at development status, define measures such as how long an operation took, time to first token and how many tool calls an agent made. Those are the right measures for engineering, and none of them says whether an answer was right or whether the user gave up. Observability tools add evaluation on top: the documentation of one open-source tool, Langfuse, lists model-graded scoring of production traces, human annotation and end-user feedback. What to watch, and who sees it, is still a product decision.

At Tcules, this is what we would put in front of the people accountable for an AI feature, drawing on Samy's method and the two cases above. It is our judgement, not a standard.

Two of these, repeated questions and the language users use, come straight from Samy's account. The fate of each output is covered in more depth in our article on why users stop using an AI feature after trying it once.

Some of this belongs in front of customers as well. Microsoft's guidelines paper (CHI 2019) asks products to help users understand how often the AI system may make mistakes. For a business customer's administrator, a plain view of how the feature has behaved recently can do that better than a per-answer rationale. This is an inference from the guideline, not something the guideline prescribes.

Where the shift holds, and where it does not

Samy's conclusion holds for the team accountable for the feature. NIST's framework asks that an AI system's functionality and behaviour be monitored in production (MEASURE 2.4) and that post-deployment monitoring plans include ways of capturing and evaluating input from users (MANAGE 4.1). A team that spends its effort on per-answer explanations and has no view of behaviour over time has the order wrong.

It does not hold for the person deciding about one output. Observability arrives later and in aggregate; it cannot tell someone whether the figure in front of them is right. That person still needs something at the moment of decision, and the research above says it should be something checkable rather than a narrative.

It also does not hold in regulated decisions. Under Article 86 of the EU AI Act, a person affected by a decision that a deployer takes on the basis of the output of a high-risk system listed in Annex III (critical-infrastructure systems excepted), where the decision has legal effects or similarly significantly affects them in a way they consider adverse to their health, safety or fundamental rights, has the right to obtain from the deployer "clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken". That is an explanation of the decision, not of the model's internals, and the same regulation requires providers of high-risk systems to run post-market monitoring (Article 72). Products in these high-risk uses need both, and the audit trail that connects them is a subject of its own.

Samy's other advice from the episode fits here too: build the logic yourself and put AI in the right place, rather than hand everything to one agent. A product whose steps are its own can explain its decision in terms of those steps, even when the model inside one step cannot explain itself.

Where to start

A rule for choosing where to invest first:

1. If a person must stand behind each output (a claim they will repeat, a figure they will file, a record the system will change), start with what they can check at the moment of decision.
2. If outputs flow at volume without anyone reading each one, or you depend on a model provider that updates its models, start with observability for the team.
3. If both are true, design them together. What users do with each check (open the source, edit the field, discard the answer) is also a direct observability signal.

At Tcules, we work on both sides of this in our AI Trust, Control and Recovery service: what a person can inspect, what they can do when the output is wrong, and what the product shows when something has gone wrong and nobody knows yet. The cheapest first input is the feature's wrong answer, its uncertain one and its refusal.