Insights
Who can audit a SaaS onboarding flow, how to find the step where new users drop out, and what a useful audit hands back. A practical guide for product teams. Your own product manager and analyst can audit your SaaS onboarding flow, and they should start it: they hold the funnel data and the context.
Your own product manager and analyst can audit your SaaS onboarding flow, and they should start it: they hold the funnel data and the context. Bring in an outside practitioner when your team disagrees about why users stop, or when nobody has walked the flow as a brand-new user in each role.
A funnel will show you the step where new users stop. It will not show you why. A good onboarding audit does both jobs, keeps them separate, and ends with a decision someone owns.
Three kinds of auditor, and what each is good for
Who should run the audit depends less on skill than on distance from the product and access to evidence.
| Who audits | Strong at | Weak at | Choose when |
|---|---|---|---|
| Your product and analytics people | The event data, the definition of "activated", the history of past changes | Seeing the flow as a stranger would; they know how it is supposed to work | You have working analytics and a specific suspicion to test |
| Your designers, researchers, support and customer success staff | What users say and where they get stuck, in their words | Counting: anecdotes are vivid and not a sample | You have no usable funnel yet, or the numbers and the complaints disagree |
| An outside practitioner | Distance from the team's assumptions; time set aside to follow one flow end to end, role by role | Context and history; they need access, test accounts and a named person who can answer questions | The team disagrees about the cause, or has audited itself and wants a second reading |
| Automated checks and tools | Repeatable surface problems, such as missing labels or broken states | Cause, consequence and priority; a long list of plausible issues is not a finding | As an input to any of the above, never as the whole audit |
Two cautions apply to all four. An audit is only as good as the evidence it was allowed to see, so the access list matters as much as the auditor. And an outside auditor does not need your source code unless the question depends on implementation, such as how an invitation is actually validated.
What an onboarding audit has to answer
There are three questions, and they need different evidence.
- Where do people stop? This is a measurement. A funnel, a cohort table or a log answers it.
- Why do they stop there? This is a diagnosis. It needs someone to watch, walk or ask, and it can come out as "unresolved".
- What should the team do, and who owns it? This is a decision. Some findings call for a local fix, some for more evidence, and some for leaving things as they are with a reason.
An onboarding review can be strong on the first question and quietly skip the second. The result is a ranked list of drop-off percentages that says nothing about what to change. The rest of this guide takes the questions in order.
How to find where new users drop out
Six checks make the difference between a funnel you can act on and one that misleads.
- Fix the finish line before reading the funnel. Name the action that means a new user got what they came for, and write it down. Be careful about the shortcut of picking whichever early action correlates most with retention. A correlation in your own data can come from the kind of customer who was always going to stay (our illustration). The chapter on observational causal studies in Trustworthy Online Controlled Experiments sets out. Treat the activation event as a working hypothesis.
- Count the right unit. In B2B, the person who signs up may not be the person who does the work. Amplitude's documentation on account-level reporting makes the point that group-level reporting suits conversion processes that involve several people. If an admin creates the workspace and a colleague completes the first real task, a per-user funnel shows two incomplete journeys instead of one complete one.
- Match the conversion window to how your customers actually work. In Amplitude's funnel documentation, the conversion window is the longest a converting user can take to finish the funnel, with a default of one day. A product where setup waits on a security review or a data import will log slow successes as drop-outs under a one-day window. Look at the time-to-convert distribution before choosing one.
- Compare absolute losses, not just percentages. The same documentation says that to find the largest drop-off the relevant comparison is the absolute decrease, not the percentage decrease. A step that loses half of 40 people matters less than one that loses a fifth of 4,000.
- Split by who arrived. Role, plan, company size and acquisition source can tell different stories. A blended funnel can hide the fact that one group stops at the first screen. This is our recommendation rather than a documented rule; check that each split still has enough people to read.
- Test the instrumentation by walking the flow yourself. Do the flow once with an event log open. A missing or double-firing event can look like a cliff in the funnel, and it is cheaper to rule out than to redesign around.
What you now have is a location, not a cause. Write it down as "most new accounts that start setup never reach the first completed task, and the loss concentrates at step X", and stop there.
How to find why they stop
Walk the flow as a new user, once per role. A cognitive walkthrough is a structured method for this: it evaluates learnability for new or infrequent users. At each step the evaluator asks whether the user will try to achieve the right effect, notice the correct action, connect it to what they are trying to do, and see that progress is being made, as summarised by the Usability Body of Knowledge from the UX Professionals' Association. The same source lists its limits: it does not estimate how often a problem occurs or how severe it is, and it depends on the evaluator's skill. Use it to generate candidate causes, then check them against data and real users.
Walk the awkward paths too. An audit that covers only the main route can miss the places where users fall off. Check at least:
- an invited teammate arriving into a workspace someone else set up;
- an expired or reused invitation;
- the empty state before any data exists;
- a user whose role cannot do the step the guide tells them to do;
- single sign-on, email verification and the second session, when the user returns a day later.
Check the sign-up and return-login steps against the accessibility criteria that touch them. Two WCAG 2.2 criteria are relevant. Redundant Entry (3.3.7, Level A) says information a user has already provided in the same process must be auto-populated or available to select, with exceptions for essential cases, security, and information that is no longer valid. Accessible Authentication (Minimum) (3.3.8, Level AA) says no authentication step may require a cognitive function test, such as remembering a password, unless it offers an alternative, a mechanism that helps (such as paste or a password manager), object recognition or personal content. Its own guidance says it covers signing in to an existing account and does not cover creating the account, so it applies to the day-two return rather than the first sign-up. That the second session is where it bites is our reading, not the standard's wording.
Use support tickets and recordings, with care. Support themes show what users say is wrong. Session recordings show what they did. If you use recordings, check the masking first. A 2017 Princeton study of seven session-replay vendors found that the services collected form input before the user submitted it, showed little default redaction of displayed page content, and in at least one case leaked a password typed into a registration form. Vendor tooling has changed since, so verify your own configuration instead of assuming either result.
How much evidence is enough
For the qualitative part, the old "five users" rule needs care. Virzi's 1992 experiments found that four or five participants uncovered about 80 per cent of the usability problems in the cases he studied, and that the most severe problems tended to turn up early. A 2003 study by Faulkner sampled random sets from 60 test users and found that sets of five revealed anywhere from 55 to 99 per cent of the problems, while sets of 10 never fell below 80 per cent and sets of 20 never fell below 95 per cent. The sensible reading is that a small round of sessions is a good way to find problems and a poor way to prove you have found them all. A fair audit says how many sessions it drew on and names what that sample cannot show.
For the quantitative part, small B2B volumes limit what you can prove. By the standard two-proportion approximation, our own arithmetic, detecting a rise in activation from 40 to 50 per cent with 80 per cent power at the usual 5 per cent significance level needs roughly 385 new accounts in each group. A team seeing a few hundred sign-ups a month cannot test every change that way. For an obvious defect, such as a step that fails for a whole role, fixing it and watching the funnel may be proportionate. Reserve formal tests for changes where the answer is not obvious and the volume exists.
What the audit should hand back
Ask for a record you can challenge, not a score.
| Part of the record | What it should contain |
|---|---|
| Scope and sources | Which flow, which roles, which data and sessions were used, and what was not examined |
| Funnel definitions | The activation event, the unit counted, the conversion window and the segments, so the numbers can be reproduced |
| Findings | What was observed, where the evidence came from, how strong it is, and the consequence |
| Cause | A stated cause, or an honest "unresolved" with the competing explanations and the next evidence needed |
| Owner and action | One of: correct, investigate, redesign, retain on purpose |
| Validation | What to check after the change, and by when the answer should be visible |
A single finding should read like this. The example is fictional and not a client result.
Observed: In a test account created as an invited teammate, the first screen after accepting the invitation is an empty dashboard. Nothing says what the admin has already set up or what this person should do first.
Consequence: A teammate who cannot see a first task may close the tab, and the admin's setup effort never turns into shared use. The empty screen is observed in the inspected flow. How often invited users reach it, and how many leave there, is not established by this sample.
Next action: The product owner pulls the invited-user funnel by role, then watches three or four invited users take the flow, to decide whether the fix is a first-task prompt, a change to who is invited at which point, or neither.
Validation: Invited users should be able to name their first task without help, and the invited-user completion rate should be compared before and after on the same definition.
If an audit cannot produce findings in this shape, you have a list of opinions. Tcules' guide to the SaaS UX design audit describes the stages of an audit, from stakeholder interviews to the audit report.
Before you ask anyone to audit it
Whoever does the work, these preparations decide how much they can find:
- [ ] Test accounts for every role a new user can arrive as, including invited teammates and a restricted role.
- [ ] Read access to the funnel, with a short note defining the activation event and any events known to be unreliable.
- [ ] A written statement of the decision this audit supports, for example "should we fix the invitation step or rebuild setup?"
- [ ] The last several months of support themes about setup and first use.
- [ ] Recordings or interview notes you already hold, with a check that sensitive fields are masked.
- [ ] A named person on your side who can accept a finding, investigate it or implement the action.
- [ ] A list of what you have already decided not to change, so the audit does not spend time on it.
Where a Tcules audit fits
Onboarding is one workflow, and the Tcules UX and Product Audit is built for a product team that can see friction but cannot agree on its cause. Tcules follows representative work across the interface, the product model and the implementation evidence, records whether each conclusion is observed, reported or inferred, and returns a finding-to-decision ledger with an owner and a validation recommendation for each action. The audit page asks you to bring product access, intended users, current concerns, known evidence, representative data, relevant analytics or support themes and the decision the audit must inform.
To start, describe the one flow that causes the most trouble and the decision your team cannot yet make. The audit page says Tcules can then bound the sample and state what the audit can credibly establish. If you already know what needs to change, commission that work instead of another audit.