Insights
What an AI agent did on a SaaS admin screen in our small hands-on test, the design rules we drew from it, and an agent-readiness check for your product. An AI agent operates your product in one of two ways.
An AI agent operates your product in one of two ways. It looks at screenshots and clicks by position, or it reads the page's accessibility tree, the browser's list of every control with its name, role and state, and acts on named elements. Many agents combine the two. Each way gets stuck on different things.
In our own small test, an agent reading only the accessibility tree could not archive an invoice behind unnamed icon buttons, pick a date from a click-only calendar, reorder a drag-only list or find a member in a long list with no search. The same agent working from screenshots got through all four, at a cost of up to 25 actions for a task that took 4 with a search box. A version of each pattern built for agents worked in both modes.
Most of that version is accessibility work your human users already need: controls with names, state the software can read, a way to do without dragging, errors written in text and saves that say they happened. Below are the test, what it showed, the design rules we draw from it, and a check you can run on your own product.
This page is about someone else's agent using your interface. For the agent you build into your own product, see our agentic UX design principles.
The test: one admin screen, nine patterns, two kinds of agent
We built a fictional admin console, "Northwind Ops", with nine patterns that are common in B2B SaaS. Each pattern appears twice: a typical version, as many products build it today, and an agent-friendly version, built to be easier for an agent. The screen has member roles, invoices, workspace seats, API keys, webhooks, a renewal date, an approval order, per-member security settings, invitations and notification settings. The page records what really changed in a hidden state object that nothing on screen shows, so we could compare what the agent did with what it said it did.
The agent was Claude Sonnet 5.5, which we operated through the Claude desktop app's built-in browser. It never saw the page source or the hidden state, and the page carried no hints. It ran twice, in separate sessions:
- Screenshot mode: screenshots, clicks and typing by screen position only. This is how general computer-use agents work: Google's documentation for Gemini says "Using screenshots, the model can 'see' a computer screen", and describes Computer Use as a Preview capability that "may contain errors and security vulnerabilities" (Google).
- Tree mode: the accessibility tree only, read through the same app's built-in browser tools, with clicks on element references and no screenshots. Other tools work this way too: Microsoft's Playwright MCP server, for example, "enables LLMs to interact with web pages through structured accessibility snapshots, bypassing the need for screenshots or visually-tuned models" (Microsoft). Our run did not use it.
We fixed the 18 task wordings, their order and the scoring before the first run. Tasks came in a shuffled order, so the two versions of a pattern were never adjacent. For each task we recorded whether the hidden state reached the target with nothing else changed, what the agent claimed, whether it claimed success it had not achieved, and how many actions it reported.
The limits, once. This is one model, one run per mode, a synthetic screen, a small browser window (870 by 374 pixels) and a tester who also built the screen. Action counts are the agent's own. All 18 tasks ran in one session per mode, so the agent could carry what it learned from one version of a pattern to the other: it saw the named credit-note icons before the unnamed invoice icons with the same layout. Two runs were voided for set-up faults and are not counted: in the first screenshot attempt, a scaled viewport made clicks land off target, so the agent changed nothing and stopped; in the first tree attempt, the page had not reset from the previous run. The tree our harness returned also left out the on and off state of switches, which the page provides; Playwright-style accessibility snapshots do expose checked state (Playwright), so our switch findings may not carry over to tools such as Playwright MCP. So the test shows patterns and direction, not rates, and other agents may behave differently.
What happened, pattern by pattern
The task was achieved unless marked otherwise. In each pair, a is the typical version and b the agent-friendly one; the role menu is named by control. Action counts are the agent's estimates; the agent's claim follows when it was not a plain "done".
| Pattern | Version | Screenshot agent | Tree agent |
|---|---|---|---|
| Role menu | custom drop-down in a scrolling box | 6 actions | 3 actions |
| native select with a label | about 18 actions; its pop-up never appeared in screenshots | 2 actions | |
| Row icon buttons | a: icons with no names | 2 actions | Not done: refused, could not tell archive from delete |
| b: the same icons with names | 2 actions | 1 action | |
| Edit button | a: shown only on hover | 6 actions | about 9 actions; claimed "unsure", could not read the value back |
| b: always visible | 6 actions | 3 actions; claimed "unsure", same reason | |
| Destructive action | a: one-click revoke, five-second undo, near-duplicate name | 3 actions, right key | 1 action, right key |
| b: type the name to confirm | 8 actions | 3 actions | |
| Date | a: click-only calendar, target four months away | 9 actions | Not done: day cells not in the tree |
| b: native date input | 6 actions; typing failed, key presses worked | 2 actions | |
| Reorder | a: drag only | 2 actions | Not done: no route without dragging |
| b: up and down buttons with names | 3 actions | 2 actions | |
| Find a member | a: list loads more as you scroll, no search, a lookalike name earlier | about 25 actions | Not done: the target never loaded |
| b: search box | 4 actions | 3 actions; claimed "unsure", switch state not shown | |
| Failure | a: red border and a note far below | reported "not sent" | reported "not sent" |
| b: inline alert naming the cause and the fix | reported "not sent", quoted the fix | reported "not sent", quoted the fix | |
| Slow save | a: "Saved" shows for 1.5 seconds | waited for and saw "Saved"; claimed "unsure" because the "Back to members" link gave no visible feedback | claimed "unsure", gave no cause |
| b: lasting "Saving…" then "All changes saved" | done, but distrusted the status | done |
What the test showed
1. In this test, the agent-friendly versions worked for both kinds of agent. The agent completed every one of them in both modes. Working from screenshots, it also completed every typical version, several at a much higher cost. Working from the tree, it could not complete four typical versions: the unnamed icons, the calendar, the drag-only list and the list with no search.
2. The same control can be easy for one kind of agent and impossible for the other. Drag-and-drop took the screenshot agent 2 actions and was impossible from the tree. The native select took the tree agent 2 actions and the screenshot agent about 18, because its pop-up never appeared in the screenshots (most likely because the browser draws it outside the page); only typing the option's letters and pressing Return worked. Anthropic's documentation names this weakness: "Some UI elements (such as dropdowns and scrollbars) might be tricky for Claude to manipulate using mouse movements", and it suggests keyboard shortcuts (Anthropic). You cannot design for "the agent". Google's guidance says modern agents combine the tree with screenshots and asks sites to "provide clean signals across all these channels" (web.dev). Our test is a concrete case of why.
3. When the agent could not be sure, it stopped instead of guessing. Faced with three unnamed icons, one of which might delete an invoice, the tree agent refused. Across 36 task attempts, neither agent claimed a success it had not achieved. Do not rely on that: Anthropic's own documentation says Claude "sometimes acts as if its actions succeeded without explicitly checking their results". The cost showed up instead as doubt. Five tasks that had succeeded came back "unsure": the tree agent could not read back two saved seat values or a switch's state, and on the slow save, the screenshot agent saw "Saved" but was unsure because the link it clicked next gave no visible feedback, while the tree agent gave no cause. In a product, doubt means retries, an agent that asks the user to check, or a support ticket. The switch case is partly our harness, as noted in the limits; we did not establish why the tree agent could not read the saved seat counts back.
4. Content that appears only on demand cost the most. The list that loaded more members as you scrolled, with no search, cost the screenshot agent about 25 actions and defeated the tree agent, which reported that the list stopped growing at about 50 members. The click-only calendar defeated the tree agent too.
5. Some risks we built in did not trigger. Lookalike names ("staging-old" and "staging-old-2", Priya Nair and Priya Naidu) caused no wrong actions, and a failure shown mainly by a red border was still read correctly. Both agents read carefully. One run each is not a guarantee, and the rules below still cover these patterns.
Design rules, each tied to what we saw
Each row starts from an observation in our test. The last column is what an outside source says; it is not something our test proved.
| Pattern | What happened in our test | The rule | What outside sources say |
|---|---|---|---|
| Icon-only buttons | Tree agent refused to choose between archive and delete; the screenshot agent managed, but it had just seen the same layout with names | Give every control a name that includes its object: "Archive credit note CN-78" | WCAG 4.1.2: name and role "can be programmatically determined" (W3C) |
| Drag-only reorder | 2 actions by screenshot; impossible from the tree | Add named "Move up" and "Move down" buttons beside the drag handle | WCAG 2.5.7: dragging must be achievable "by a single pointer without dragging" |
| Click-only calendar | 9 actions paging four months by screenshot; day cells absent from the tree | Accept a typed date: a native date input or a labelled text field, with the calendar as an extra | WCAG 2.1.1: all functionality operable through a keyboard |
| Long list with no search | About 25 actions by screenshot; target never loaded from the tree | Put search or a filter above any list a task has to find one item in | No WCAG criterion requires search; this is a usability rule |
| Switches and saved values | Three tree results came back "unsure" because the agent could not read the state or value; for the switch, our harness's tree omitted state the page provides | Expose state: a native checkbox, or `role="switch"` with `aria-checked`; show the saved value as text | WAI-ARIA 1.2: a switch's `aria-checked` "indicates whether the input is on" (W3C) |
| Hover-only actions | No difference by screenshot; about 9 actions against 3 from the tree | Keep row actions visible, or reachable by keyboard focus | WCAG 2.1.1 |
| Errors | Both versions read as "not sent"; only the inline alert let the agents quote the cause and the fix | Write the error in text by the field, with the cause and the fix, in an alert region | WCAG 3.3.1, 3.3.3 and 1.4.1 (colour alone is not enough) |
| Slow saves | With the 1.5-second "Saved", the screenshot agent waited for it and saw it, but was unsure because the next link gave no visible feedback; the tree agent was unsure and gave no cause. A lasting "All changes saved" was distrusted because it showed before any change | Keep the status until the next change, name what was saved ("Digests per week saved: 3"), and make every link and button show that it did something | WCAG 4.1.3 status messages; OpenAI tells builders to "check the saved result before reporting success" (OpenAI) |
| Destructive actions | Both agents picked the right key and webhook; typing the name cost more actions | Name the exact target in the confirmation; offer undo that lasts longer than five seconds | OpenAI: "Keep users in control of ... destructive changes"; Gemini can mark an action as needing the user's confirmation |
| Native select | About 18 actions by screenshot; 2 from the tree | Keep the native select; do not swap in a custom widget to suit screenshot agents | Anthropic: dropdowns "might be tricky"; use keyboard shortcuts |
Two notes on the rules.
A typed confirmation is not a human check. An agent can type the webhook's name as easily as a person. What the typed name does is make the agent state the exact target, which guards against the lookalike that sits next to it. If an action needs a person's judgement, the agent platforms put that confirmation on their side: OpenAI's guidance keeps users in control of destructive changes, and Gemini returns a "require_confirmation" decision for the builder's app to show the user. Your product's job is to make the target unmistakable in what the agent passes on.
Rules from other sources that our test did not cover. Google's guidance adds four we did not test: avoid transparent overlays over interactive elements, use real buttons and links rather than styled `div` elements, link labels to inputs with the `for` attribute, and keep layouts stable so that a button is not in a different place on each page (web.dev). Treat those as advice from the source, not findings of ours.
Which rules also help human users
Almost every rule above is an accessibility requirement first. Names, roles and states are what screen readers read. An alternative to dragging serves people who cannot drag precisely. A typed date and visible row actions serve people who use the keyboard. A text error and a lasting save status help everyone, and a status in a live region reaches screen reader users without moving their focus. Google's guidance puts it in one line: "Everything we suggest to make a site 'agent-ready' also makes sites better for humans."
That is the business case. If accessibility work is already on your roadmap, agent-readiness rides on it, and the reverse also holds: a team fixing its product for agents is closing accessibility gaps at the same time. If you want the human side in more depth, our inclusive product design guide covers it.
The one place the two pulled apart in our test was the native select: the most accessible control was the costliest for the screenshot agent. We would side with the people and the standard, keep the native control, and let the agent use the keyboard, as Anthropic's documentation suggests. Two rules also go beyond accessibility. Search on long lists is good usability rather than a WCAG requirement. Naming the target in a confirmation is a safety rule.
When to give agents a proper door instead
An agent clicking through your interface is a workaround for a missing route. If many of your customers hand agents the same task, you can give the agents a direct route to it:
- An API or an MCP server lets an agent call the action itself, with its own permissions and logs, instead of finding a button.
- WebMCP is a draft from a W3C community group that lets a web page offer its actions to an agent as tools (WebMCP draft). It is not a W3C standard, and Google says it "is in active development and may change", with an origin trial in Chrome.
- MCP Apps works the other way round: your tool returns an interactive interface that renders inside the agent's chat (MCP blog). Our article on generated interfaces covers what that asks of a design system.
None of these replaces the screen. People still use it, and agents will reach every task that has no door. Our test screen had no sign-in. Whether agents may sign in, pass multi-factor checks or get past bot defences is a policy decision for you and your customers, not a layout question.
Run the check on your own product
A quick pass, before any agent runs. Open the accessibility tree in your browser's developer tools on the five screens your customers use most, and check:
- Every button, link and field has a name that says what it acts on.
- Every switch, checkbox and tab shows its state.
- Every drag has a button alternative.
- Every date field accepts a typed date.
- Every long list a task has to search has a search or filter.
- Every error is text next to its field, with the cause and the fix.
- Every save leaves a status that lasts until the next change and names what was saved.
- Every destructive action names its exact target before it runs.
Repeating our test. It needs a small test account and an agent you can run in both modes.
- Choose five to ten tasks your customers would delegate. Write the exact wording, and what success looks like in your database, before any run.
- Keep the ground truth where the agent cannot see it: your database or audit log, never the screen.
- Run the agent twice, once with screenshots only and once with the accessibility tree only, in separate sessions.
- Reset the data before each run, and check that it reset. One of our runs was void because it had not.
- Use the browser window at its natural size. A scaled viewport voided another of our runs.
- For each task, record the truth, the agent's claim, any success claimed but not achieved, and the number of actions.
- Read every "unsure" and every refusal as a design finding, not an agent failure.
Where you fix something, run the same tasks again and compare.
The rules for the person who directs your own agent are in our human-in-the-loop patterns and our AI product readiness checklist. This page is the other side: your product as the thing an agent operates. If you want an outside view, our UX product audit follows representative workflows, roles and states through a product, and the checks on this page can be part of the sample we agree.