Insights
To design UX for an AI product, first settle the user decision, model role, sources, failure, human control and evaluation. A checklist with next steps. Designing UX for an AI product starts before the interface.
Designing UX for an AI product starts before the interface. Settle six things first: whose decision or work gets better, what the model may do on its own, what its output rests on, what happens when it is wrong, who can correct it, and how you will judge it on real scenarios. Where any of these is still open, the next step is research, a prototype or a narrower scope rather than screens, and sometimes the honest answer is a rule, better search or a workflow change instead of AI.
Each item below gives the question, what "ready" looks like, and what to hold back until it is.
The checklist at a glance
Work through it for one AI interaction at a time, not for an AI strategy, and with the people who would have to defend the decisions. At Tcules, we mark each answer as decided (written down, with an owner), assumed (everyone thinks they agree) or open. Assumed answers drift apart quietly, so treat them as open.
| # | Question | Ready when | If it is not, do this next |
|---|---|---|---|
| 1 | Whose decision or work gets better? | You can name the person, the decision and what they do today without AI | Research the work; or choose a non-AI fix |
| 2 | What may the model do on its own? | Its role, its reach and the point where a person commits are written down | Define the scenarios, tools and authority |
| 3 | What does the output rest on? | Sources are named per output and visible when the person decides | Settle data access and provenance first |
| 4 | What happens when it is wrong? | Each failure has a visible state and a way forward | Prototype the difficult and partial states |
| 5 | Who can correct it, and in time? | The person with authority sees the evidence before the consequence | Design the review and recovery path |
| 6 | How will you judge it? | Representative scenarios, criteria and failure classes are agreed | Build the evaluation set before expanding |
1. Whose work gets better, and would something simpler do it?
The question. Who is trying to decide or accomplish what, what do they do today without the AI, and what does a good outcome look like in their terms rather than the model's?
Ready looks like. A short written statement naming the person, the decision (approve a claim, choose an intervention, close a record), today's way of doing it and the outcome that would count as better. It also says why a deterministic rule, improved search, clearer information architecture or an ordinary workflow change would not meet the need. "Summarise faster" is not enough when the real job is to make and defend a decision.
Not yet. Do not choose a model or build a general assistant while this is open.
The evidence. Google's People + AI Guidebook is blunt that a rule or heuristic "will work just as well, if not better" in many cases, and lists where AI is probably not better: where predictability matters most, where errors are costly, and where complete transparency is required. Google's Rules of Machine Learning, by Martin Zinkevich, open with Rule #1: don't be afraid to launch a product without machine learning. Apple's Human Interface Guidelines on generative AI say generative AI is "not the right solution for every situation". The NIST AI Risk Management Framework opens its MAP function, which establishes the context for an AI system, with the same step: document the intended purposes and the specific users and their expectations (MAP 1.1), and define the business value (MAP 1.4).
Next move if not ready: research the work and define the decision, or conclude that AI is not the answer.
2. What may the model do on its own?
The question. Does the model retrieve, classify, recommend, generate or act? Which records can it read or change? What needs a person's explicit commitment, which role is accountable when it acts, and what is it explicitly not for?
Ready looks like. For each output, a chosen level of autonomy on a ladder such as: show information, suggest, prepare something for approval, act and report, act silently. A named accountable role (a role, never "the AI"). A written list of what is out of scope. A plan for telling people when they are dealing with AI. If you offer the product in the EU, Article 50 of the AI Act, applicable since 2 August 2026, requires providers to design systems that interact directly with people so that those people are told they are interacting with an AI system, unless that would be obvious to a reasonably well-informed person given the circumstances and context of use, and at the latest at the first interaction. Whether and how it applies to your product is a question for legal advice; Apple's guidance makes the same point as a design matter: clearly identify when and where you use AI.
Not yet. Do not allow open-ended action, and do not rely on a prompt alone to set the boundary.
The evidence. Microsoft Research's Guidelines for Human-AI Interaction (Amershi and colleagues, CHI 2019), 18 guidelines tested by 49 design practitioners against 20 AI products, begin with "make clear what the system can do" and include "scope services when in doubt". Apple's guidance advises against automating destructive or hard-to-undo actions and asks for confirmation before significant actions taken on someone's behalf. Anthropic engineers Erik Schluntz and Barry Zhang, in Building effective agents, recommend finding the simplest solution possible, which "might mean not building agentic systems at all". If the feature is an agent, setting authority per action is covered in agentic UX design principles.
Next move if not ready: define the scenarios, tools, objects and authority (the product model) before any interface.
3. What does the output rest on?
The question. Which data may be used, who is authorised to provide it, and which sources does each output depend on? Can the person see that basis at the moment they decide?
Ready looks like. Sources listed per output type; access that matches the permissions of the person asking; provenance a person can open and check; and a defined state for missing, partial or conflicting sources. Where personal data is involved, permission and data use are settled before design starts.
Not yet. Do not design the output screen before you know what the person will be able to inspect beside it.
The evidence. NIST's Generative AI Profile (NIST AI 600-1, July 2024) names confabulation, confidently stated but false content, as a risk specific to or made worse by generative AI. Apple's guidance is to avoid asking a model for factual information unless it has access to verified, up-to-date information for the task. The design consequence is that the basis of the answer is part of the product, not a debugging aid. Provenance and state are taken further in AI product design as a context architecture problem.
Next move if not ready: settle data access, sources and provenance; for a chat-style assistant, see answering from the customer's own data.
4. What happens when it is wrong?
The question. What does being wrong look like here, concretely (the wrong customer emailed, the wrong figure in a report, the wrong record closed)? How would anyone notice? What does the product do when information is missing, retrieval is incomplete, a tool call fails or an answer is plausible but unsupported? And if a wrong output has already been sent, saved or acted on, how is it contained?
Ready looks like. Named bad outcomes in the user's terms; a visible state and a route forward for each failure; the difficult and partial states prototyped, not just the happy path; and, where the AI is complementary rather than essential, a way to finish the task without it.
Not yet. Do not treat a happy-path demo as evidence that the product works.
The evidence. Apple's guidance notes that it is easy to prototype an exciting generative feature quickly but hard to make it work in all real-world situations, because small changes to inputs, or the same input given twice, can produce very different results. Where the AI is not essential to the task, it suggests considering a non-AI fallback when possible. The Microsoft guidelines ask products to "make clear how well the system can do what it can do", which is only possible once you know how it fails. When people try a feature once and stop, one of the possible causes sits here, and why users stop using an AI feature after trying it once shows how to tell the causes apart.
Next move if not ready: prototype the difficult and partial states before building the full feature.
5. Who can correct it, and in time?
The question. Who can inspect, correct, reject, escalate, undo or retry, and do they have the evidence and the authority to do it before the consequence becomes hard to reverse?
Ready looks like. The check sits at the point of consequence, not at the end of the run. The reviewer sees the evidence the output rests on. Rejecting or editing is as easy as accepting. An override records the corrected state and a reason, so human judgement stays visible.
Not yet. Do not add an approve button or confidence wording and call that control.
The evidence. A person in the loop is not a safeguard by default. In a CHI 2021 study, Bansal and colleagues found that AI explanations "increased the chance that humans will accept the AI's recommendation, regardless of its correctness". The Microsoft guidelines ask for efficient correction and dismissal, and Apple's for Edit, Undo, Retry or Adjust near generated content. At Tcules, we cover the patterns themselves in human-in-the-loop UX patterns and in nine products that place the human check at the point of consequence; for one consequential output, our AI Trust and Control Canvas works through it block by block.
Next move if not ready: design the review, override and recovery path; for a feature already live, audit evidence, correction and recovery before adding anything.
6. How will you judge it?
The question. Which representative scenarios, criteria and failure classes decide whether the product is useful and safe enough to advance, and who signs that off?
Ready looks like. A scenario set drawn from the real work, covering typical, edge and adversarial cases; criteria written in terms of the user's decision, not only model output; named failure classes; a review owner; and an acceptance test that decides the next step.
Not yet. Do not optimise a single generic quality score, and do not expand across roles and use cases on the strength of one narrow success.
The evidence. OpenAI's evaluation best practices advise task-specific evals that reflect real-world distributions, with test data that includes typical, edge and adversarial cases. Apple's guidance adds testing out-of-scope, vague and poorly phrased requests. The NIST framework treats this as a formal decision: a determination of whether the system achieves its intended purpose and "whether its development or deployment should proceed" (MANAGE 1.1).
Next move if not ready: build the evaluation set and failure classes; once evidence and control are enough for a narrow case, release that case in a controlled way.
Reading the result: which next move
Expect a mix of ready and open answers. At Tcules, our AI Product UX Readiness Assessment ends in a disposition, a recommended next move, and the same idea can earn different ones at different moments:
| Where the open answers cluster | Next move | Hold back on |
|---|---|---|
| The user decision (item 1) | Research the work and define the decision | Choosing a model or a general assistant |
| The model's role (item 2) | Define the product model: scenarios, tools, objects, authority | Open-ended action |
| Failure (item 4) | Prototype a bounded scenario, including the difficult states | Treating a demo as the product |
| Evaluation (item 6) | Build the evaluation set and failure classes | One generic quality score |
| Nothing material for one narrow case | Build a controlled release | Expanding to every role at once |
| An existing feature people find hard to rely on | Improve evidence, correction and recovery | Adding confidence wording without changing control |
Deferral is a legitimate outcome, and so is deciding that AI is not the right intervention at all.
Two projects where a gate decided the next step
At Tcules, we worked on both of these projects, and we describe them only as far as their case pages go. They show readiness decisions being made; they do not show production performance.
ConnectX arrived with personas, KPIs, feature ideas and AI-generated concepts, but no product structure connecting a request to data and interface behaviour. In our reading against this checklist, the model's role (item 2) was the open question. The project team defined eight representative scenarios, covering system status, forecasting, asset performance and operational conditions, each specifying the request and detected intent, the data and modules required, the digital-twin behaviour and the next action, before the coded demonstration was treated as coherent. This was demonstration work on an interim data layer; it did not demonstrate live telemetry, adoption or savings.
BuildTwin had the opposite shape. The AI could run structural-drawing checks, but accountability stayed with the quality engineer, so, in our reading against this checklist, items 3 and 5 shaped the design: each result showed its drawing reference, source document, observation and conclusion; when information was missing, the interface showed what the reviewer needed to add before running the check again; and an override required a corrected state and a reason. This is a product model, not a shipped outcome: the available material does not establish what shipped, and we claim no adoption, time saved or accuracy improvement.
Run the checklist on your own feature
Tcules is a digital product design studio creating ambitious new software products and making established ones better. We offer two ways to run this checklist. Our AI Product UX Readiness Check is a self-assessment for one AI feature: twelve questions on purpose, role and limits, basis and evidence, and failure and recovery, to use before committing engineering time and again before release. It does not evaluate your model, measure accuracy, assess your data or cover security, privacy or regulatory obligations; it covers the product interaction around the AI.
When the decision needs an outside view, our AI Product UX Readiness Assessment examines the product contract before a team commits to a larger build, across six gates that the items above follow, and leaves a product-decision statement, a model-role and authority boundary, representative scenarios, review and recovery responsibilities, evaluation criteria and one recommended disposition. If the idea passes and you need the product designed, that is our AI product UX work.