Designing AI tutors: what the 2026 trials show

Design an AI tutor around the moment a student gets something wrong, and keep the student doing the thinking. That is the clearest lesson from the large randomised trials published in 2026. A guard-railed tutor built into practice helped middle-school students recover from mistakes and produced a small, marginally significant gain a week later. A well-known tutor placed one click away barely got used, and its schools' gains resembled what the same practice platform delivers without AI. Both tutors were built to coach rather than answer. What differed was when and how the help reached the student.

An AI tutor that answers every question is a homework machine. The evidence below shows what a tutor that teaches does instead, and the product decisions that follow: when the tutor offers help, how hints escalate, how it avoids giving answers away, how progression works, what teachers see and how to design for students who will not ask.

What the 2026 trials tested

Three randomised trials carry most of the weight, and two of them come from the same school district in Tennessee.

NUMI: AI support crossed with mastery. Oreopoulos, Liut, Sungu and Low (EdWorkingPaper 26-1552, August 2026; also NBER w35621) randomised more than 6,000 students in grades 6 to 8 in Hamilton County Schools. Each student was assigned to a practice platform with or without an AI tutor, and to mastery or open progression. The session lasted about 50 minutes in March 2026, with a short test one week later. The findings:

• Pace and accuracy. Students with the tutor attempted fewer questions and moved more slowly, but answered more accurately when they did attempt.
• Recovery after mistakes. Among mastery students, the tutor raised the chance that the next attempt after a mistake was correct by 8.5 percentage points and cut the attempts needed to get back to a correct answer by 0.96. It also added 2.88 minutes of clock time on the way.
• The delayed test. Students with the tutor and the mastery rule scored 40.2 per cent on the practised Exercise 1 question, against 37.0 per cent without the tutor. The authors call this marginally significant (p = 0.065) and suggestive. There was no difference on unpractised material.
• Mastery on its own. The rule raised the share of students reaching three correct answers in a row by 28.7 percentage points, and did not by itself improve the delayed test.

Khanmigo: access without use. Oreopoulos and Low ("One Click Away", NBER w35620, also EdWorkingPaper 26-1551) ran a two-year cluster randomised trial in 18 Tennessee middle schools. Students used Khan Academy with Khanmigo, configured to coach rather than give answers, during daily remedial maths sessions. Achievement rose by 1.3 national percentile ranks per term, about 0.06 to 0.08 standard deviations a year, which the authors note resembles gains from Khan Academy practice without AI. The authors' explanation is engagement, measured in the second year:

• 96 per cent of students messaged the tutor at least once.
• The median student sent no messages on two thirds of practice days.
• The median student messaged it in only 17 per cent of exercise sessions that contained a mistake.
• Only 14.5 per cent of messages contained a mathematical question or a step of reasoning; most were bare answers or clicks on suggested prompts.

Within the treated group, achievement tracked skills brought to proficiency, not chats, a descriptive association rather than a causal one, the authors stress.

Human support: more use, no reading gain. Robinson, Gormley, Trindade Ribeiro and Loeb of Stanford University (EdWorkingPaper 26-1451, June 2026) ran two randomised trials with 355 elementary students on an AI literacy platform. Despite scheduled time, nearly half the students working alone never used it, and average use was 2 to 5 minutes a week. An in-person tutor focused on engagement rather than instruction raised use by 1 to 4 minutes a week and stories read by 71 to 80 per cent. Reading achievement did not improve. This is the null result a product team should keep in view: more engagement, still too little to move learning.

Two earlier studies frame these. In a PNAS study of nearly a thousand high-school students (Bastani and colleagues, 2025), an unrestricted GPT-4 interface improved practice grades by 48 per cent, then cut grades by 17 per cent once access was removed. A version with prompts designed to protect learning largely avoided that harm. In an exploratory trial in five UK secondary schools, Google's LearnLM team and Eedi had expert tutors supervise every AI-drafted message. The tutors approved 76.4 per cent with no or minimal edits, and students guided by the model did at least as well as those chatting with human tutors on every outcome measured.

Design choices against the evidence

The pattern across the rows: structure decides whether the tutor's help is used, and used at the right moment. Model capability is a precondition, not the design.

Put the help where mistakes happen

The NUMI authors describe the mistake as the moment when tutoring should matter most. A student who has just answered wrongly can guess again, read a worked solution, ask for help or give up. In NUMI, a wrong answer triggered a guided walkthrough without the student asking; after later mistakes it alternated with solution review. In the Khanmigo trial, help was a click away and students mostly did not click. The Khanmigo paper reports that Khan Academy switched the tutor to open by default shortly after page load on 29 January 2026, and later rolled out a broader redesign that activates it throughout practice, adopted in response to low student-initiated use.

For a product team, the decision is when the tutor speaks without being asked. Our reading of the trials:

• Trigger on the wrong answer. This is the event with the clearest measured benefit. Do not wait for a student to type a question they may not know how to form.
• Offer a start, sparingly. NUMI's "Help me get started" button posed a two-option reasoning question before any attempt, then disappeared for the next two attempts. That rationing is worth copying: help before an attempt should not become the default first move.
• Stay quiet when things are going well. A student answering correctly does not need interruption, and every interaction costs time.
• Treat time as a cost you design for. NUMI's tutor made recovery more productive, not faster. In a fixed class period that meant fewer questions reached. The authors suggest shorter, more targeted post-error interactions and heavier support where a student is likely stuck or an item is especially diagnostic.

A hint-escalation pattern

The steps below are our pattern. Steps 1 to 3 follow what NUMI built and tested together as one treatment; steps 0 and 4 are our additions for a production product, not trial findings.

1. Attempt first. The student works the problem without help. The tutor is visible but silent.
2. First-step question, on request. The tutor names the kind of first step needed and asks a two-option question: one correct move, one plausible misconception. The student chooses, then gets a short explanation either way and goes back to the problem.
3. After a wrong answer, a guided walkthrough. The tutor names the likely misconception and leads one step at a time. Each step ends with a check the student must answer (yes or no, A or B, multiple choice or a short answer), with a skip always available.
4. Worked solution with explanations on demand. Once the solution is shown, the student can pick any step and get an explanation of the idea behind it, not a restatement of the text.
5. Escalate to a person. Repeated mistakes on the same skill, or a student skipping every walkthrough, become a flag on the teacher's view rather than more hints.

The pattern never ends in the tutor giving the final answer inside the chat. The worked solution in step 3 is the platform's own, shown after an attempt, and the next question still has to be solved alone.

Stop the tutor giving the answer away

A prompt that says "do not reveal the answer" is not a design. NUMI's appendix describes the layers the team built around the model:

• Screening every incoming message. A classifier flags crisis messages, which receive a fixed help-line response, and off-topic messages, which are redirected. It also reads intent, separating frustration, a calm request to move on, profanity and answer-seeking, each with its own policy.
• Checking every reply before it shows. A critic scores each draft for correctness and informativeness, an improver rewrites weak drafts, and a separate validator grades the student's answers to the tutor's checks.
• Keeping replies short. One or two short message bubbles, guiding by reasoning.

Build answer-seeking into the product as a handled state, not a failure. A student demanding the answer is telling you something: they are stuck, tired or out of time. NUMI's response was a supportive redirect to the problem and a visible skip. The Bastani trial shows what happens when nothing handles that state: the tool becomes a crutch, and performance falls once it is gone.

Ground the tutor in the exercise on screen. NUMI's tutor sat beside the problem and worked from its solution steps, which is what made it possible to check replies for correctness at all.

Mastery gates and progression

A useful negative finding from NUMI is that a mastery gate alone did not produce learning a week later. Three correct answers in a row can come from understanding, but also from repetition, lucky streaks or guessing until a sequence works. The NUMI authors conclude that a short-run streak should not be treated as durable understanding.

The gate did matter in combination. It gave students a reason to engage with help after a mistake, because skipping past the explanation left them stuck. That is the design argument for gating: it creates demand for the tutor at the right moment. The costs are real, so design the gate deliberately.

• Pair the gate with support after errors. A gate with only a worked solution produced more attempts, not more learning.
• Mix the evidence of mastery. The authors suggest varied item types, a brief explanation requirement, a confidence rating before feedback or a delayed retrieval check before declaring mastery.
• Watch exposure. Students who spend longer clearing early material reach less of the later material in a fixed session.
• Show the learner where they are. Progress the learner cannot see feels like being held back. When we redesigned a creative-writing learning platform for students aged 8 to 18, we reorganised content so learners could see their current activity and their next step, with progress indicators along the way.

What teachers see

The trials say little directly about teacher dashboards, so this checklist is our design guidance, not a finding. It follows from two results: in the Khanmigo trial the median student messaged the tutor in only 17 per cent of sessions with a mistake, and in the Stanford trials an in-person tutor alongside the platform (programme staff in one district, middle-school students in the other) raised use where scheduled access alone did not. A teacher view should turn what the tutor noticed into a decision the teacher can act on in the lesson.

• Who is stuck now: students with repeated mistakes on the same skill, in real time.
• Who is not using help: students who skip walkthroughs or never open the tutor after a wrong answer.
• Common misconceptions: the misconceptions the tutor named, grouped across the class, so the teacher can reteach one idea to many students.
• Mastery against retention: gates cleared next to delayed-check results, so a streak is not mistaken for learning.
• Pace: how far each student has got, since tutor support slows students down.
• Answer-seeking and frustration: how often each student asked for the answer or signalled frustration.
• Safety events: crisis messages and the response shown, routed to the right adult under the school's policy.
• The conversation, on demand: the full exchange behind any flag, so the teacher can judge it.

For a wider view of what learners and administrators need from a learning platform, see our article on eLearning platform UX.

Design for students who will not ask

In the Khanmigo trial the tutor sat inside mandatory session time with a teacher present, and it offered suggested prompts. The median student still sent it no messages on two thirds of practice days. The authors argue that engagement responds to design, through defaults, structure, facilitation and interface, and that this is a behavioural problem as much as a technical one.

What that means for a product team:

• Make help the default after an error. Do not make the student open it. Struggling students are the least likely to seek help unprompted, the Khanmigo authors note, citing earlier research.
• Ask questions the student can answer. Two-option and short-answer checks lower the effort of engaging. Typing a question about your own confusion is harder.
• Design the human role. The Stanford trials raised use with in-person tutors running check-ins and reflection. The LearnLM trial put an expert tutor over every AI draft. Decide whether your product supports a teacher, a tutor or neither, and design the screens for that person.
• Measure engagement at the moment that matters. Report the share of mistakes followed by real use of help, not logins or total messages.
• Expect usage to be a ceiling. In the Robinson trials, even improved use stayed far below what the provider recommends. A tutor nobody uses has no effect to measure.

Sidebar: rules for chatbots used by minors

California now regulates companion chatbots used by minors, and a tutor product may fall within its definition. The state's SB 243 (Chapter 677, approved 13 October 2025) applies to "companion chatbots": AI systems with human-like responses that can meet a user's social needs and sustain a relationship across interactions. Its exclusions cover bots used only for customer service, business operations, productivity, internal research or technical assistance; tutoring is not named either way. For a user the operator knows is a minor, the law requires:

• disclosure that the user is interacting with AI;
• a default reminder at least every three hours of continuing interaction to take a break, and that the chatbot is not human;
• reasonable measures to prevent sexually explicit visual material or statements urging such conduct.

It also requires a published protocol for responding to suicidal ideation or self-harm, with referral to crisis services, and annual reporting from 1 July 2027. A tutor that remembers a student, encourages them and chats across sessions starts to look like the definition. NUMI's fixed help-line response to crisis messages shows the kind of protocol a tutor needs regardless.

Read this sidebar as design input for the product team, not as legal or regulatory advice. Whether SB 243 or a similar law covers your tutor is for your legal and regulatory owners to decide.

What the evidence does not show yet

• Short horizons. The NUMI result comes from one 50-minute session and a four-question test a week later. It says nothing about grades, state tests or a year of use.
• Small effects. Most NUMI students still got the delayed question wrong. The Khanmigo trial could not separate the tutor's contribution from structured practice.
• One district, mostly maths. Two of the three main trials ran in Hamilton County schools, and two of the three were in maths.
• Specific products. NUMI's authors warn against generalising to all uses of language models in education: effects depend on interface, prompts, timing, content quality and incentives.
• Engagement problems are not solved. The Stanford trials raised engagement and still found no reading gain.

The direction is consistent, though. Where the tutor met students at their mistakes and kept them reasoning, it added something. Where it waited to be asked, it was seldom asked. That is a design decision, and it is made long before a model is chosen.

We work on learning products for edtech teams; you can see how on our EdTech and learning expertise page.