How to use a UX product audit so it ends in decisions

Insights · Guide

A UX product audit is worth its cost when it changes a decision your team has already named, and when every finding that survives review leaves with a chosen action, one owner and a check that shows whether the action worked. Much of that is in the buyer's hands: the question you give the audit, the evidence and access you bring, how you read the findings and what you do in the fortnight after the report lands. Finding problems can be the cheaper part, and AI tools are making it cheaper still. In a field study of one development project, Morten Hertzum found that the major cost of its usability evaluations was fixing the problems, not finding them.

This guide is for SaaS product and engineering leaders who are about to commission an audit, or have just received one. It follows the audit in the order you meet it.

Before you commission: name the decision and the person who makes it

Evaluators outside software learned this long ago. Michael Quinn Patton built utilisation-focused evaluation on the premise of "intended use by intended users": an evaluation is judged by whether named people use it for a named purpose, and his checklist for the approach includes a step to simulate use of the findings before any data is gathered. Applied to a product audit, that becomes two things to write down before you sign.

The decision, in one sentence. For example: "By the end of next month, our head of product decides which three workflow problems enter next quarter's roadmap." Other decisions a SaaS team might hand an audit:

• whether the roles and permissions model can carry the larger customers you are now selling to;
• whether a module needs a redesign or deeper modernisation of the data and rules underneath it;
• whether an AI feature is clear and recoverable enough for wider release;
• why one workflow keeps generating support contacts after several rounds of fixes.

A rehearsal. Invent three plausible findings and ask the decision owner what each would change. If the answer to all three is "nothing, we would carry on", the audit is pointed at the wrong question, and it is cheaper to discover that now than after the readout. The rehearsal also tells you who must attend the readout: anyone who would have to approve or fund the response.

If you are still comparing providers, the questions to ask them are in our guide to choosing a UX audit provider. If onboarding is the decision, the onboarding audit guide covers the evidence that question needs.

Bring the evidence the screens cannot show

A review of the visible interface can show that something is inconsistent or hard to use. It cannot show who is affected, how often, what it costs or why it happens. Much of that evidence may already sit inside your company, and the audit can only use what you hand over.

• Access as each role. Test accounts for every role that touches the workflow, including administrators and the most restricted users. In SaaS products, a problem can live in what one role can see or do that another cannot.
• Realistic data. An anonymised or seeded account with real volume and awkward cases: empty states, thousands of rows, long names, failed imports, expired permissions. A demo account with five tidy records hides them.
• Product analytics for the workflow, with a note of what is not tracked or is known to be wrong.
• Support themes. The top contact reasons for the workflow and a handful of raw tickets, which carry the customer's own words.
• Sales and customer success notes: lost-deal reasons, escalations during onboarding, renewal objections.
• Research you already hold, however old or informal, so the audit does not repeat it.
• People. Someone who knows the business rules, and an engineer when the question depends on how the product is built.
• Constraints. What cannot change: contracts, regulation, platform limits, a release freeze.

If you have already run checks yourself, for example with our SaaS UX audit checklist, send the results too, including the items you have decided not to fix.

Decide one more thing before handing material over: whether customer data, recordings or screenshots may be processed with AI tools, and under what terms. Ask the auditor to confirm in writing how they will follow that decision.

Using an AI-generated issue list

You may meet AI in two places: a list you generate yourself by running a model over screenshots, or an offer whose findings are mostly produced by software. Either way, treat the output as a list of candidate issues.

The evidence supports that framing. Peitong Duan, Jeremy Warner, Yang Li and Bjoern Hartmann built a Figma plugin that used GPT-4 to run heuristic evaluation, tested it on 51 interfaces and with 12 expert designers, and reported that its feedback was useful for catching subtle errors and improving text, but less useful over repeated iterations, and that the designers described several uses for it despite its imperfect suggestions. Our comparison of audit methods sets out the recent studies of how much such tools find and how often they are wrong.

A practical way to use one:

1. Run it before the audit starts, on the workflow in scope.
2. Fix the obvious items yourself, and mark the ones you already know about or have decided to leave.
3. Hand the annotated list to the auditor as one input among the evidence above, so their time goes on cause and consequence instead of rediscovering surface issues.

What a generated list cannot supply is the part that decides your roadmap: how often your customers meet the problem, what it costs them, whether the cause sits in a business rule or the data model, and what to do about it. Those answers need your evidence and logged-in access. Judge any audit, automated or not, by the decisions it supports, not by the length of its list.

When the report arrives: read each finding for its evidence

Before ranking anything, sort what each finding rests on. Three kinds of evidence are all legitimate, but they warrant different actions.

The same discipline is written into the US intelligence community's analytic standards (ICD 203), which require products to distinguish underlying information from assumptions and judgements, to explain uncertainty, and to name indicators that would change a judgement. A good finding does the same in a few lines. Ask of each one:

• Which role, state and data was it seen in?
• Is the cause observed, or proposed? If proposed, what else could explain the symptom?
• What would change the conclusion?
• Where does the sample stop? A finding from one workflow may or may not apply to the others.

Treat severity ratings as a starting judgement. In the same paper, drawing on earlier studies and his own experiment with usability inspections, Hertzum concluded that agreement among individual evaluators' severity ratings is so low that it is risky to base decisions about system revisions on one evaluator's ratings; he recommends involving several evaluators and suggests ratings from users as another option. You hold the evidence a single auditor lacks, so re-rate the findings you might act on with product, support and engineering together, using your own data on reach and consequence.

Ask for a proposed direction on the findings you intend to act on. In a study at a large Danish job portal, Kasper Hornbæk and Erik Frøkjær found that four members of its development team rated redesign proposals as more useful in their work than descriptions of usability problems: the proposals gave them new ideas for problems they already knew about, though none wanted to receive only problems or only redesigns. One study of one product is limited evidence; in our judgement, a problem statement starts a debate, and a proposed direction gives the team something to test.

An illustrative finding, read this way

Illustrative example, not a client finding.

Notice what the finding does not claim: how many customers are affected, or what the stalls cost. The evidence does not reach that far, so the decision begins with a check rather than a redesign.

Decide in one working session, not by circulating the PDF

Hold one session soon after the readout, with the decision owner, the product lead, an engineering lead, someone from support and, ideally, the auditor. Work through the findings in order of consequence and give each one a disposition.

Three rules keep the session honest.

• One named person owns each decision, not a team. They are the one who reports back on validation.
• Write the validation check before the work starts, while nobody is invested in the answer.
• Sequence by consequence, confidence and dependency. An impact and effort grid is a fine way to order work, but a high-impact guess and a high-impact observation land in the same square. Keep confidence beside impact, and do structural work before the local fixes it would overwrite.

Then move the decisions into the tool where the team plans work, each linked back to its evidence. A PDF is where findings go to wait.

A simple measure of whether the audit made a difference comes from Paul Sawyer, Alicia Flanders and Dennis Wixon, who defined an impact ratio: the share of reported problems that receive a fix. They reported an average of 78% across ten commercial products their own group inspected, so treat that as a figure from one team's practice, not a benchmark. A finding accepted or deferred with a written reason is also a decision; what should worry you is a finding nobody decided anything about.

After release: check the decision, not just the screen

Run the validation checks you wrote down. Where a metric is the right check, choose it from the finding's consequence rather than from whatever the dashboard already shows. The HEART framework from Kerry Rodden, Hilary Hutchinson and Xin Fu at Google is a useful way to do that: it maps product goals to signals and then to metrics across happiness, engagement, adoption, retention and task success. For the illustrative finding above, the signal is task success when removing a person, and contacts about stalled approvals, not overall engagement.

Two cautions. A B2B workflow may have too few users for a clean experiment; if so, a before-and-after comparison needs a note on what else changed in the same period. And a check that shows no improvement is a finding too: in our judgement, it is a reason to look again at the diagnosis, not only at the screen.

Revisit the deferred findings at each planning cycle and ask whether their triggers have fired. When the next question arises, decide whether it needs another audit at all. If the team already knows what has to change, the useful next step is the design or build work itself.

Where the Tcules audit fits

Our UX Product Audit is built around the same sequence. It starts from the decision the team cannot yet make, agrees the sample of workflows, roles and states before review and states where that sample cannot support a broader conclusion. Each finding records the observation, source, affected condition, consequence, confidence, diagnosis, decision, owner and validation, and says whether a conclusion is observed, reported or inferred. The resulting decision queue separates quick corrections from structural work and names what should stay unchanged. AI-assisted analysis can widen what is inspected, we verify consequential conclusions against their sources, and your own restrictions decide whether product material can be used with AI tools at all.