How should an AI product let users choose a model?

Let people compare models on their own input when they are making a lasting choice, choose the model automatically when they would rather not choose at all, and in both cases show which model answered and keep a way to override it. A bare list of model names with benchmark scores asks users to read evaluation metrics, which Eden AI found its developer users did not. Eden AI, an API platform for AI models, tried all three approaches in turn, and each one moved the work of judging to a different place.

This article draws on the first episode of Beyond the Interface, the Tcules podcast, in which I spoke with Samy Melaine, founder and CPTO of Eden AI. The conversation was recorded around November 2024 and published in February 2025, so read his remarks as a record of that moment. Eden AI is a Tcules client. The Tcules work for Eden AI, described in the workflow creation case study, was on workflow creation and search, not on the model comparison or routing features discussed here.

Three ways to hand over the choice

Samy described the first two steps in one sentence:

"we started by giving our users the mathematical metrics to compare different models and then we moved to visual comparison which was way better" Samy Melaine, recorded around November 2024 (40:10)

The third step, choosing for the user, came next. Each approach answers the same question, "which model should handle this?", but asks a different party to judge.

Comparing by metrics

Eden AI began by showing evaluation metrics for each model. Its users were developers, and that turned out to be the problem, as Samy explained, along with what Eden AI did about it:

"developers were not used to comparing AI models … they're not used to checking the mathematical evaluation metrics of different models, they don't talk that language that data scientists talk … we had a lot of questions about model selection coming from our users and we understood that the best way to actually let them compare the models and understand which one to pick is to make it visual for them … our users are able to … select any file in their computer and compare different models visually" Samy Melaine, recorded around November 2024 (23:33)

There is a second weakness, which Samy raised later in the conversation:

"output is really different depending on the input so each AI is trained differently so maybe for your use cases another AI would work or another model would work better than for my use cases" Samy Melaine, recorded around November 2024 (39:10)

A score measured on someone else's data says little about yours.

Public rankings built from human votes have the same limit. Chatbot Arena, now LMArena, ranks models from crowdsourced votes on users' own prompts, which tells you what many people prefer on their prompts, not what works on yours. Its method has also been challenged: The Leaderboard Illusion (Singh and colleagues, 2025) reports that undisclosed private testing let a few providers try several variants and publish only the best score.

The trade-off: metrics are precise and cheap to display, but they shift the burden onto the user's ability to interpret them, and they describe a workload that is not the user's.

What to design: use metrics to narrow the field, not to decide it. Facts a user can check, such as price, context window, supported file types and region, are safer filters than a single quality score. Where a quality score appears, say where it came from and when it was measured. In the picker itself, describe each model by what it is good for, its speed and its cost, not by a version name alone.

Comparing on the user's own input

The key detail in Samy's answer is whose input goes in: any file from the user's own computer, not a vendor's sample. He gave two examples from that period. For audio:

"whether it's a transcription of an audio if you want to compare two models that transcribe your audio you can see the differences between the transcription very visually" Samy Melaine, recorded around November 2024 (24:47)

For an image or video analysis task, the user can open their own video and see what each model detected. He also described developers sharing their workflows with product and business colleagues to check prompts and output (28:05), because the person who judges the output may not be the person who integrates it.

The trade-off: comparison on the user's own data is the most persuasive evidence a user can get, but it has costs. My reading of the method, not a claim Samy made:

• Every comparison runs several models, so the user waits and pays for each of them.
• One file is an anecdote. A choice made on a single sample can fail on the rest of the workload.
• Judging by eye works where differences are visible, as in transcripts or detection boxes. Differences in open-ended text are subtler, and a familiar model name can sway the verdict. Chatbot Arena handles that bias by showing two anonymous answers and revealing the models' identities only after the vote.

What to design:

• Accept the user's own file or prompt, and make it easy to run a small set rather than one.
• Align the outputs so differences stand out: a diff for text and transcripts, overlays for detections.
• Put cost and response time beside each output, because the best-looking answer may not be worth its price at volume.
• Offer to hide model names until the user has picked, where brand loyalty could decide the result.
• Let the user share the comparison with the colleague who judges quality, and turn the pick into the configuration in one step.

Choosing automatically

Visual comparison helped, but it did not remove the reluctance, and the plan Samy described was to choose per request:

"we still feel that our users still don't like choosing a model and users are pushing us more and more into giving them our recommendation … we are building a recommendation system that for each input you give we'll select on the fly the best model to handle it" Samy Melaine, recorded around November 2024 (40:23)

He added what the user would no longer have to do: "if any new model comes in then it will be selected if it suits your use case so we get that model choice out of your mind" (41:01). When I asked how Eden AI communicates its confidence in what it recommends, he answered with two commitments: the choice stays optional, and the monitoring shows what was chosen.

"if you prefer to trust your choice or your own model you can do it but if you let us choose then we are very transparent and we give you in the monitoring what models are [chosen] for your use case and you can see how it fluctuates" Samy Melaine, recorded around November 2024 (41:36)

Earlier in the episode he had put the principle more broadly. As models grow larger, explaining a single output gets harder, so, in his words, "observability now is more crucial than explainability" (35:38).

A far larger public test of this approach came later. OpenAI introduced GPT-5 on 7 August 2025 as a system with a real-time router that decides which model answers "based on conversation type, complexity, tool needs, and your explicit intent". The next day Sam Altman wrote that the autoswitcher had broken for part of launch day, and that OpenAI would "make it more transparent about which model is answering a given query". On 12 August 2025 he announced that users could choose between "Auto", "Fast" and "Thinking", and that GPT-4o was back in the picker for paid users. Altman added that "most users will want Auto"; the manual choice came back beside it.

The trade-off: automatic choice removes a decision that, on Samy's account, Eden AI's users did not want to make, and it can pick a different model for each input and take up new models as they arrive. In return the product owns the outcome. The same request may be answered by a different model next week, which matters wherever output has to be reproducible or audited. And the reason for a choice is hard to state honestly, because the ranking step can itself be a trained model, as OpenAI's router is.

What to design, drawing on what products that route automatically document today:

• Show which model answered, every time. GitHub Copilot shows the model on hover in chat and at the end of a cloud agent's response; OpenRouter's Auto Router returns it in the response's `model` field.
• Let users set the bounds. OpenRouter takes a list of allowed models. Copilot offers tiers in plain words (Efficiency, Balance, Intelligence) for trading cost against quality, rather than asking users to weigh models.
• Keep the manual choice next to Auto, and let a user pin a model for work that must behave the same way each time.
• Keep a conversation on one model unless the task changes; OpenRouter documents this as session stickiness.
• Tell people when the pool changes. Microsoft's Guidelines for Human-AI Interaction ask products to "update and adapt cautiously" (G14) and to "notify users about changes" (G18). A router that quietly adopts a new model is exactly the kind of change both guidelines address.
• Explain at the level you can stand behind. The same guidelines ask products to "make clear why the system did what it did" (G11). For a router, an honest account may be the bounds and the objective ("the cheapest model in your shortlist that suits this request") plus a record over time, rather than a confident reason for each answer.

Where Eden AI's automatic choice stands now

The recommendation system Samy described as in progress did ship. By March 2026 Eden AI's documentation described a special model name, `@edenai`, that asks Eden AI to choose the model per request. The last version of that page, from August 2026, let developers restrict the choice to a shortlist, set a scale from best quality to lowest cost, and opt in to a report of what was chosen and whether it had been ranked, was the only candidate, or was a default. That report, in the page's own words, "reports the outcome, not the machinery".

On 1 September 2026 Eden AI removed the page; the commit says the router "is no longer documented", and the old address now redirects to provider routing. Today's documentation describes a narrower automatic choice: the developer names the model, and Eden AI chooses which provider serves it, by lowest cost unless asked to favour speed, latency or instruction-following. That choice never changes which model answers. It is "invisible by default", and request metadata reports which provider served a request when the developer asks for it. The documentation does not say why the per-request model choice was removed or whether it still runs.

Which approach fits your product

These are judgements drawn from the evidence above, not rules any of these companies has published:

• Who chooses, and how often? A developer choosing once for a feature needs comparison on their own data and a way to name or pin the result. An end user choosing per message is better served by an automatic default with the model shown and an override within reach.
• Can the user judge the output by looking? If yes, side-by-side comparison carries most of the weight. If quality differences are subtle, add a blind comparison or a small evaluation set built from the user's own cases.
• Must the output be reproducible or auditable? Then pin a model version, and allow automatic choice only within limits the user declared and the product records.
• Do candidate models differ widely in cost? Then expose the trade-off in plain words rather than as a model list.
• Can you show which model answered, on every answer and over time? If not, you are not ready to choose automatically.

Where this work sits

Model choice is one decision inside the wider relationship between people and model behaviour. For the generative features around it, see generative AI in SaaS; for where a person should review or approve output, see human-in-the-loop UX patterns and nine AI products that place the human check at the point of consequence.

Tcules designs that relationship through AI Product UX: how people understand, direct, evaluate, control and recover from model behaviour, from defining the model's role to evaluation before release. To hear Samy in full, watch the episode.