Why users stop using an AI feature after trying it once

When people try an AI feature once and then go back to doing the task by hand, three causes are worth testing before anything else: the first outputs they got were wrong; checking an output cost more than redoing the task; or the review step gave them no way to correct, partly accept or undo what the AI produced. Three quick tests on the users who stopped tell these apart: grade the outputs they actually received, time a check against a manual redo, and list what the review step lets them do. If the outputs were wrong, fix accuracy first; no interface rescues a wrong answer.

This page is the diagnosis. Patterns for building the review step itself, and examples of products that make outputs easy to check, have their own pages, linked where they apply.

Three causes, and the test that separates them

The causes can occur together. All three tests are cheap enough to run on the same small sample.

First, rule out the cases that are not failures

Some flat usage lines are not a design problem, and treating them as one wastes the effort.

The task is rare. A feature that drafts a quarterly report will look abandoned in weekly usage. Measure repeat use against how often each user does the task, not against calendar time.

They never met the feature again. If the entry point sits away from where the task happens, users who did the task again may never have been offered the AI. Only users who were offered it at the point of work and chose the manual path tell you anything about the three causes. In our view, an empty prompt box is a weak offer too: a returning user who cannot think what to ask has not really been offered the feature at the point of work.

The manual way was already quick. If the task takes a minute by hand, even a correct output that is easy to check may not be worth switching for. That is a question about whether the feature should exist, not about how it is designed.

What is left, people who do the task again, are offered the AI at that moment, and do it by hand anyway, is the group the three tests are for.

Cause 1: the first outputs were wrong

A demo runs on inputs chosen to work. A user's first run uses their own inputs, which may be messier, longer or stranger than anything in the demo set. If those first outputs were wrong, stopping is a reasonable response.

Two lines of research suggest how heavily an early error weighs. Dietvorst, Simmons and Massey found that people who watched an algorithm make forecasts became less willing to rely on it, even when it outperformed a human forecaster, because they lost confidence in it faster than in a person who made the same mistake (Journal of Experimental Psychology: General, 2015). Nourani and colleagues varied the order in which people met an AI system's strengths and weaknesses; those who met weaknesses first relied more on themselves, which meant fewer errors, but underestimated what the system could do (IUI 2021). Both studies are about forecasting and video recognition tasks rather than product features, but they point the same way: what a person sees first shapes how much they rely on the system afterwards. A demo shows strengths first by design. A user's first real input may not.

The test. For the users who stopped, retrieve the outputs from their first few real runs and have someone who knows the task grade each one as correct, partly correct or wrong. Grade a sample from users who stayed in the same way. If the users who stopped mostly received wrong outputs, this is an accuracy problem. If they mostly received correct outputs and stopped anyway, accuracy is not the cause, and you are looking at cause 2 or cause 3. In our judgement this is the most useful single test on the page, because it decides whether the fix belongs to the model or to the product.

What to fix. The model, the data it sees, or the scope. Narrowing the feature to the inputs where it is reliably right can be quicker than improving the model. Then tell users how often it gets things wrong, which is the second of Microsoft's guidelines for human-AI interaction, make clear how well the system can do what it can do.

Cause 2: checking costs more than redoing

A user who is answerable for the result has three options when an output appears: accept it unchecked, check it, or do the task themselves. If the only way to check is to repeat the work, the AI has added a step instead of removing one, and for anyone who has to defend the result, doing it by hand is the cheaper honest option. That is our reading of the trade; the research below supports the part about how people weigh the check.

Vasconcelos and colleagues ran five studies with 731 participants who found the exit of a maze with help from a simulated AI (CSCW 2023). Their account is that people choose strategically whether to engage with an explanation, weighing its cost against the cost of doing the task or simply relying on the AI. The results followed the cost. Explanations written as step-by-step directions beside the maze did not reduce overreliance compared with no explanation. The same path highlighted on the maze itself, which is quicker to check, did reduce it in the hardest mazes. In a small exploratory follow-up, an explanation that coloured the AI's path differently after it crossed a wall brought average overreliance to zero among the 15 participants who saw it; the authors note this needs testing with more people. Raising the reward for a correct answer also reduced overreliance. The authors suggest that explanations which fail in other studies may fail partly because they do not reduce the cost of verifying the AI's answer enough.

Their measure is overreliance, accepting a wrong answer, not abandonment. What carries over is the mechanism: whether people check depends on what checking costs. Fok and Weld make the general case that explanations help decisions only to the extent that they let a person verify the AI's answer, and that most tasks do not allow easy verification whatever the explanation method (AI Magazine, 2024).

Developers describe the same trade in surveys. In Stack Overflow's 2025 survey, 66% named "AI solutions that are almost right, but not quite" as a frustration, and 45% said debugging AI-generated code is more time-consuming (2025 Developer Survey). In the 2026 survey, asked how much they trust the output of AI tools in their workflow, 48% chose "I trust it when I can easily verify the output", and 16% said they trust it for many tasks but not important work decisions (2026 Developer Survey). These are developers using coding tools, a different population from your users, but the pattern is the one this cause describes.

The test. Take a handful of real tasks. Time a careful check of the AI output, the check a user would need before they could defend the result, and time doing the same task by hand. If the check takes about as long or longer, this is your cause. A quicker version of the same test: ask what the user has to open to confirm the output. If the answer is the same documents they would have read to do the task, the check is the redo.

What to fix. Make the check cheaper than the redo, without changing the model. As an illustration, not a client case: a feature reads a supplier contract and fills the field "Payment terms: Net 45". With nothing beside the value, confirming it means finding the payment clause in a twelve-page PDF, which is the manual task. If the value links to the clause, opened and highlighted, the check becomes reading one sentence. Same model, same output; only the cost of checking has changed. For products that make outputs checkable this way, see nine AI products that place the human check at the point of consequence.

Cause 3: the review step never hands control back

Some review steps offer only two actions: approve or dismiss. The user cannot change the value, accept three of five fields, or undo after committing. A reviewer who sees an output that is nearly right then has two moves: approve something they do not fully believe, or reject all of it and do the task by hand. For a careful user, the second move ends the feature's usefulness.

The strongest evidence that control matters comes from Dietvorst, Simmons and Massey's follow-up work (Management Science, 2018). Across three studies, people were considerably more willing to use an imperfect algorithm when they could modify its forecasts, even when the modification allowed was tightly restricted, and the size of the allowance made little difference. People who could modify the forecasts were more satisfied with the process and more likely to choose an algorithm for later forecasts. Microsoft's guidelines put the same requirement as support efficient correction: make it easy to edit, refine or recover when the AI system is wrong.

The test. Follow one item from suggestion to the moment it becomes binding, and list every action the user has at that moment. Can they edit the output in place, so that their edited version is what commits? Can they accept part of it? Can they undo after committing, and for how long? If the only actions are approve and dismiss, this is a cause whatever else is true.

What to fix. The commit step. Approve, edit and reject as equal outcomes, review by exception, and undo windows are set out in human-in-the-loop UX patterns. If the AI acts in other systems rather than filling a field, authority, stopping and recovery need to be decided per action, as in design principles for AI agents that act on a user's behalf.

Why a confidence score alone does not fix causes 2 or 3

A confidence score tells the user how sure the model is. It does not make this output any cheaper to check, and it gives the user no way to change it. Adding one to a feature with an expensive check or an approve-or-dismiss step leaves both problems where they were.

The research is consistent with that. Zhang, Liao and Bellamy found that showing confidence helped people calibrate their trust in an AI, but that calibrated trust alone did not sufficiently improve the decisions people made with it (FAccT 2020). Bansal and colleagues, in studies with 1,626 participants on three tasks, found that explanations did not improve team accuracy beyond simply showing the AI's confidence, and that explanations increased the chance that people accepted the AI's recommendation whether or not it was correct (CHI 2021). Some of their participants turned the score into a rule: of the 90 who mentioned using confidence, 27% described an explicit threshold below which they were likely to distrust the AI, with reported thresholds mostly between 80 and 100 and as low as 65.

A score still has a job. It can tell a user which outputs to check first. It works alongside a cheap check and an editable result; it does not replace either.

What to fix first

1. If the users who stopped mostly received wrong outputs, fix accuracy or narrow the scope before anything else.
2. If their outputs were mostly correct and a check takes as long as the redo, fix the check.
3. If the check is quick but users can only approve or dismiss, fix the commit step.
4. If both 2 and 3 apply, our judgement is to start with the check when the user is answerable for the result, because they have to verify before an edit means anything; and to start with the commit step when errors are obvious at a glance but tedious to correct by rejecting and redoing.

What to log so the next diagnosis is quicker

• What happened to every output: accepted unchanged, edited then accepted, partly accepted, dismissed, or replaced by manual work afterwards. Opens and clicks cannot separate the three causes; the fate of the output can. In a study of GitHub Copilot, the rate at which shown suggestions were accepted predicted developers' perceived productivity better than measures of how long completions stayed in the code (Ziegler et al., MAPS 2022). That is one coding tool, not your feature, but it is a reason to log what users did with each output rather than only whether they opened the feature.
• The input and the output together, so a sample can be graded later. Without them the test for cause 1 cannot be run.
• Time from output shown to commit or dismissal. It is a rough proxy, but long times that end in dismissal point towards an expensive check.
• Repeat use against task frequency, so a rare task is not read as abandonment.

Where this work sits

Deciding what an AI feature may produce, what a person can inspect and how they correct or recover from it is the work of our AI Product UX, which we at Tcules define as the wider product relationship: how people understand, direct, evaluate, control and recover from model behaviour.