Comparative usability testing: build both design directions to the same depth

Test both directions with users only when the choice is structural, hard to reverse and genuinely contested. If you do, build both to the same depth: the same tasks, the same states (including errors and timeouts) and the same hardest case, so the difference a participant notices is the model and not how much work went into each prototype. For a cosmetic choice, the extra prototype work is not worth it.

A tempting shortcut is to polish the favoured direction and show the other as a sketch. That comparison measures the build, not the idea.

Why a lopsided comparison misleads

A prototype that is less complete than its rival can be judged on the wrong things, and the error can run in either direction.

It can lose on finish. In a study of a simulated cash machine, Tractinsky, Katz and Ikar (2000) found that how attractive the screen layouts looked affected how usable people rated the system after using it, while how usable it actually was had no such effect. That study varied layouts, not prototype finish, but it shows how appearance can colour usability ratings. The evidence on visual polish is not one-sided: an earlier study by Wiklund, Thurrott and Dumas (1992), summarised by Virzi and colleagues, found that making prototypes look more realistic did not change ease-of-use ratings.

It can also win because its hard parts were never shown. In the Wiklund study, the prototypes could not reproduce the real device's slow responses, so every prototype was rated as more usable than the product itself. A sketch that skips the timeout, the error and the awkward order type spares its model from the moments that could expose it.

Either way, the result can reflect what each prototype left out rather than the model it carries. Testing one direction alone has its own bias. In a CHI 2006 study by Tohidi, Buxton, Baecker and Sellen, 48 participants tested paper prototypes of a home climate control. Each design scored higher when seen alone than when seen alongside two alternatives (significantly so for two of the three), and the authors conclude that alternatives make ratings less prone to inflation and draw out more and stronger criticism. A fair comparison of two directions is worth more than a flattering test of one.

Depth is not fidelity

Two equally rough prototypes make a fair comparison. One polished and one rough do not.

Fidelity sounds like a measure of how finished a prototype looks. Virzi, Sokolov and Karis (CHI 1996) describe several separate dimensions on which a prototype can differ from the real product: breadth of features, degree of functionality, similarity of interaction and aesthetic refinement. In two experiments they found that low- and high-fidelity prototypes surfaced substantially the same usability problems.

Depth, in this article, means the first two of those dimensions: breadth of features (how much of the product the prototype covers) and degree of functionality (how completely the details of each feature's operation, its states included, are worked out). In both experiments the paper prototypes still carried the functionality the test tasks needed, and they surfaced much the same problems as the high-fidelity versions. That suggests the level of finish matters less than matching what each prototype lets a participant do. Pick any level of finish you like, then hold it equal across both directions.

Equal depth does not mean prototyping everything. The GOV.UK Service Manual's guidance on the alpha phase suggests focusing on the areas you think will be most challenging, doing the minimum needed to test your riskiest assumptions. Keep the scope narrow; make it the same scope for both.

When two directions are worth building

At Tcules, our judgement, not a measurement, is that building both directions to the same depth comes close to doubling the prototype work for the part you compare. The cost is justified when all three of these hold:

If the choice is cosmetic (a label, a colour, a layout that changes nothing underneath), pick one, ship it and learn from use. If only one direction is credible, test that one properly.

A live A/B test answers a different question. It needs a built product and real traffic, so it can only compare options someone has already paid to build. The comparison here happens before the build, while changing the model is still cheap.

What equal depth looked like on Phoenix

At Tcules, we joined the design team for Phoenix, a decentralised exchange built by Ellipsis Labs, through Kopper for about ten months. We set out the whole decision in our Phoenix case study; here is the part that bears on testing.

Two models for the trade form were credible. Buy/Sell is market-centred: choose a direction with a toggle, pick the asset, enter an amount. In/Out is wallet-centred: what leaves the wallet carries the direction, and either asset can be changed in place. It looked like a wording choice, but it produced two different forms. In a specialist product, vocabulary is part of the architecture.

Choosing in a review meeting would have replaced an open question with the most senior opinion in the room. So both models were built as clickable prototypes, from the asset page through to order review, and prepared for user testing. Both carried filled forms, review screens, an asset changed midway, and success and timeout states. Each order type, including limit, recurring, stop loss and bracket, was designed in both models, in empty and filled states, with review, success and timeout screens behind them. That included the bracket order, which carries both take-profit and stop-loss triggers.

Stakeholders could then operate both models and follow each through a real order, instead of weighing two explanations.

What we cannot tell you: the test findings are not part of the material we hold. The live Phoenix perpetuals form leads with a Long/Buy and Short/Sell toggle, the market-centred model; in perpetuals a trade opens a leveraged position, so the wallet-centred swap form had less to offer there. Whether the spot test pointed the same way, we cannot say.

The equal-depth checklist

Before the first session, check both prototypes against each line. Any line where they differ is something a participant might be reacting to instead of the model.

• Same tasks. Both prototypes support every task in the script, from the same starting point to the same end point.
• Same states. Empty, filled, review, success, error and timeout exist in both, for every task.
• Same hardest case. The most complex variant (on Phoenix, the bracket order) is built in both, not only in the favoured one.
• Same mid-task changes. If a participant can change their mind halfway in one prototype (swap an asset, edit an amount), they can in the other.
• Same content. Identical data, amounts, names and copy wherever the models do not require a difference.
• Same finish. The same level of visual refinement, whether rough or polished.
• Same interaction. Both clickable, both on the same device, with the same responsiveness: no delay or loading state in one that the other lacks.
• Same care. Ideally the same people build both. In the Tohidi study, the same team designed all three alternatives, and the researchers tried to keep them consistent in fidelity, functionality and quality.
• Neutral names. Label them A and B, not "new" and "current" or by their advocates' names.

Running the comparison so order does not decide it

If each participant tries both directions, the order becomes a variable. Research-methods texts such as Research Methods in Psychology (Price, Jhangiani, Chiang and colleagues) name the effects to expect: practice (the second attempt goes better because the task is now familiar), fatigue (it goes worse because the participant is tired) and context or contrast (the second option is judged against the first). In a comparison of two trade forms, practice is the obvious risk: by the second prototype, a participant already knows what a limit order needs.

The standard remedy is counterbalancing: half the participants see A first and half see B first, assigned at random. That stops order from favouring one direction, and if you look at the results separately for each order, it shows whether carryover happened at all. Tohidi and colleagues presented their three alternatives in counterbalanced order.

One more caution from the same CHI 2006 study. Showing alternatives drew out more criticism, but participants still did not reliably produce constructive suggestions; in the authors' words, usability testing is "a means to identify problems, not provide solutions". Expect participants to show you where each model breaks, not to design the fix or merge the two for you. Which model to build stays a team decision, made on what was observed.

At Tcules, our Prototyping and Usability Testing work starts from the question the prototype has to answer. Where the question is whether a product model holds across states, we build a state-rich interactive prototype, then record which hypothesis was supported, challenged or left unresolved, and what the prototype could not represent.