Insights
Enterprise users are reachable through your customers. Recruit experts, test on their own data, and design tasks that run for hours, not minutes. You test enterprise software by recruiting the people who do the work, giving them their own kind of data, and setting tasks that run as long as the real work does.
The short answer
You test enterprise software by recruiting the people who do the work, giving them their own kind of data, and setting tasks that run as long as the real work does. The users are reachable through the customer. The harder part is test design: a 30-minute task on dummy data tells you very little about software people use for eight hours a day with their own records.
Ginny Redish put the problem plainly in a 2007 essay in the Journal of Usability Studies (archived copy). The typical usability test is an hour or two of short, discrete scenarios, each with a clear ending or a correct answer. Complex systems do not fit that shape, and she asked how to evaluate systems that are too complex for typical protocols. This article is our working answer for enterprise and B2B products: who to recruit, what data to use, how to build tasks, which method to pick, and how many people you need.
If the obstacle is that you cannot reach users at all, start with what to do when real users cannot take part. This page covers running the tests once you can.
Why consumer-style testing fails here
Redish describes the people using complex systems as domain experts who solve open-ended problems with a great deal of information that may be incomplete, and sometimes with no way to know at the time whether the result is right. Three consequences follow for a test.
- Goals shift. She notes that high-level goals may be vague and change as the expert works through the data, so it can be very hard to define in advance what counts as effective or efficient. People also stop when they are satisfied enough, so completion is a judgement, not an event.
- You may not know the answer. If you do not know the whole data set, you cannot say whether a task result is correct. You rely on the participant's judgement that they reached a reasonable outcome.
- Surrogates mislead. Redish defines a false positive as a product that does fine in testing but gives real users many problems, and a false negative as a product that shows many problems in testing that the real users would not have. She says false negatives may be more likely with systems like these if the participants are not domain experts.
Enterprise products add constraints of their own: the buyer can be someone other than the user, access differs by role, and the data belongs to the customer. The table sets the two approaches side by side.
| Consumer-style test | Enterprise test | |
|---|---|---|
| Participants | Anyone matching a broad profile | Experts in a specific role, frequently inside one customer account |
| Session length | One hour | Hours, or a work week, or a diary over several weeks |
| Tasks | Short, with one right answer | Open-ended, shifting goals, completion judged by the expert |
| Data | Dummy or generic | Realistic, shaped like the participant's own records |
| Permissions | One account type | Role-specific; the test must match the participant's access |
| Who else matters | The user | The user, the admin who grants access, and the buyer who signs off |
| Success measure | Task success and time | Task success, plus whether the output is trusted and usable in the next step of the job |
| Scope | The screen or flow | Components and the whole system, because only the whole shows if the problem got solved |
The last row comes from Redish: component-level tests are useful, but the only true measure is whether the user solves the problem, however large it is.
Recruiting experts through the customer
Recruit the real role, not a stand-in. Redish's questions are the right ones to settle first: how to get experts' time, how much incentive they need, and, if you must use surrogates, which characteristics of the target users you must match.
Practical routes, in the order we would try them:
- Customer success and account admins. They know who does which job and can introduce you inside an account.
- A customer advisory board, if the company has one, for the first conversations and for sponsors who can unlock others.
- Your own support and onboarding queues, for people who have recently struggled with a task you want to test.
- Prospects in a late sales stage, only with the sales owner's agreement, and kept apart from any commercial conversation.
- A recruiter or panel for roles that exist across companies, such as accountants or analysts, when the customer route is closed.
A recruiting checklist to agree before the first invitation:
- [ ] The role, seniority and weekly usage you need, written as screener questions.
- [ ] A spread across roles and permission levels, so you do not only meet administrators.
- [ ] Permission from the participant's manager or the account owner to spend work time on the session.
- [ ] An incentive the participant's employer allows. Public-sector and regulated staff may not be permitted to accept gifts or payments, so ask before you offer, and consider a donation or a product contribution instead.
- [ ] A conflict-of-interest check: do not recruit someone mid-renewal, mid-escalation or mid-negotiation without the account owner's say.
- [ ] A consent form that covers recording, screen sharing and the customer's confidentiality terms, and says plainly that the test is of the product, not the person.
- [ ] A backup participant for each slot, because expert calendars move.
Realistic data: a decision rule
Dummy data breaks enterprise tests because the product's hard problems live in volume, messiness and the participant's own knowledge. An expert recognises their own records and judges the output against what they expect, which they cannot do with invented ones. Choose the data in this order and stop at the first option that your constraints allow.
- The participant's own data in their own environment, when the customer agrees and the test can run read-only or in a sandbox copy. Best for fidelity.
- A copy of the customer's data, de-identified. Be careful with the word. The UK regulator states that pseudonymised data is still personal data and that pseudonymisation is not a route to taking data outside the law. Involve the customer's data protection lead before you copy anything.
- Seeded data built with the participant's role. Ask a domain expert to describe a realistic case, then build the data set around it: realistic volumes, awkward exceptions, half-finished records. Redish cites Caroline Jarrett's advice to go into the field and capture cases from real work to use as tasks.
- Generic sample data, only for component-level checks, and say so in the report.
Whatever you choose, record which option it was. A finding from seeded data and a finding from the participant's own records carry different weight, and your stakeholders should be able to tell them apart.
Tasks that take hours
Design the task around a job, not a feature. A job such as "reconcile this month's exceptions" or "work out why this forecast changed" has an outcome the participant cares about and a trail of sub-steps that you do not script.
- Scenario chains. Link several tasks so the output of one is the input of the next, because in enterprise products the failure can sit in the hand-off between screens, roles or tools. This is how you find what a single-screen test cannot.
- Let goals change. Give a starting question and let the participant refine it, as they would at work. Redish notes that experts also try what-if scenarios on the data.
- Accept the expert's completion call. Ask them how confident they are in the result and why, instead of marking against an answer key you may not have.
- Think aloud with care. Among the techniques Redish lists is think-aloud, "especially cued retrospective" where talking during the task would add too much cognitive workload.
- Capture usage between sessions. She lists unattended data capture alongside observation and interviews for long evaluations.
One example from the literature shows how long such a study can run. Redish summarises a study by Scholtz, Morse and Potts Steves in which intelligence analysts trained on a new system, did a practice task, and then had a week to research a question and write a report as they normally would. They were debriefed daily, and software recorded their keystrokes and links followed. Redish calls that closer to an instrumented pilot release than a typical usability test, and notes it let the team see the whole analytic process. Most product teams will not run anything that large, but the shape transfers: train, practise, then give a real job and a realistic span of time.
She also records a limit worth remembering. A Swedish train-driver project found that it was hard to build a simulated scenario complex enough to capture the drivers' expertise. If your sandbox cannot reproduce the difficulty of the job, say so, and treat what you learn as partial.
A study menu by question
Pick the method from the question, not from habit.
| If you need to know | Use | Notes |
|---|---|---|
| How people really do the job, before you design | Contextual inquiry | A field method from Hugh Beyer and Karen Holtzblatt, summarised by Nielsen Norman Group: observe and interview users in their own setting, as a master and apprentice. Suits complex systems and expert users. |
| Whether a redesigned flow works for experts | Task-based test on realistic data | Keep the chain of tasks and the expert's own judgement of completion. |
| What happens over weeks: habits, returns to unfinished work, outside tools | Diary study | Participants log activity as it happens, as Nielsen Norman Group describes, over days, weeks or longer. Expect dropouts and self-reported data. |
| Which of two designs supports the work better | A comparative test at equal depth | See how to compare two interaction models at equal depth. |
| Where an interface breaks known principles, fast | Inspection, not testing | See UX audit vs heuristic evaluation. Redish notes inspection methods are hard to use when the evaluator is not a domain expert. |
| How the product behaves with real use at scale | A controlled release behind a feature flag | Open the change to a small, agreed group of accounts, watch real use, and keep a way back. It needs the customer's agreement and an engineering team ready to switch it off. |
Two principles help. Test components when the question is local, and test the whole system when the question is whether the work gets done. And remember that no single method or measure fits all systems of this kind; Redish says so directly.
How many people you need
The famous answer is five. It comes from Jakob Nielsen and Thomas Landauer's 1993 model of how usability problems are found, and Nielsen's later article reports that a single user revealed 31% of problems on average across the projects they studied, so five users find most of them in a qualitative study. The same article says the formula assumes comparable users: for two groups it suggests three or four users per group, and for three or more groups about three each. A separate Nielsen Norman Group page recommends at least 20 users for quantitative studies.
Treat that as a floor and a heuristic, for three reasons.
- Enterprise products have several roles, so you are testing several user groups, not one.
- Five participants can mislead. In a study by Laura Faulkner, random groups of five users found between 55% and 99% of the problems from a pool of 60; groups of ten had a floor of 80%, and groups of twenty a floor of 95%.
- Long tasks produce more paths through the product, so a few participants may never touch the area that matters.
Our practical reading: plan small rounds per role, iterate between rounds as Nielsen advises, and increase the number where a role is critical or where the first round shows wide differences between people. State your sample honestly in every report: how many people, in which roles, on what data, and what you could not cover.
Running sessions across time zones and getting stakeholders to watch
Remote moderation suits enterprise work because the participant can use their own machine, tools and data. Some habits help.
- Send a setup note a week ahead: what access they need, what to have open, and what not to share. Do a technical check with the participant's IT contact if the system sits behind a VPN.
- Split long tasks into blocks with breaks, and agree a time zone and a hard stop before the first block.
- Use two people: one moderates and one takes notes, so the moderator can follow the expert's reasoning.
- Make clear what you will not ask. If the data is the customer's, tell the participant they should not show anything they are not permitted to share.
Findings from a long study are easy to ignore if nobody watched. Invite product managers, engineers and the account owner to live sessions, with a rule that observers stay silent and note what they see. Share two or three short clips of an expert struggling with a real task, played with the participant's permission. Report observations by role and by task chain, and say where the evidence is thin.
If you want help designing a study of this kind, our prototyping and usability testing and UX research services are where we work on it. For the design principles behind the products you are testing, see enterprise UX design principles for complex workflow software.