A tutor that always has an answer teaches a learner to trust fluency. A tutor that can mark the edge of its knowledge gives the learner something more useful: a reason to check the path.
Key takeaways
- Naming a boundary doubles error-catching. Learners caught an error in 61% of episodes where the tutor named its uncertainty first — versus 27% without.
- Uncertainty needs a foothold. A boundary plus a next step (a check, a source, a smaller case) produced useful follow-ups in 64% of episodes; a bare admission stalled sessions.
- Specific beats general. “As an AI, I may be wrong” was removed — it signals responsibility without telling the learner where the risk lives.
- Three signals, three moves. Fact uncertainty points to a check; meaning uncertainty to a clarifying question; example uncertainty to a smaller case.
The problem with a smooth answer
We began with a simple review of sessions where learners corrected the tutor. The visible failure was usually factual—a definition was too broad, a historical detail was wrong, or a code example quietly relied on an assumption that did not hold. The more interesting failure came earlier: the answer sounded complete enough that the learner did not think to challenge it.
In 62 reviewed sessions, learners were more likely to catch an error when the tutor had named a confidence boundary before answering. The boundary did not need to be dramatic. “I can explain the common case, but I am less certain about the exception,” was often enough to change how the learner read what followed.
This is one of the oldest problems in tutoring, and it predates the existence of language models. Human tutors face the same pressure: a confident-sounding answer is more readable than a hedged answer, and the learner is happier in the moment. The tutor who hedges looks uncertain, and an uncertain tutor is a tutor the learner is less likely to come back to. The tutor who does not hedge is, in many cases, the tutor who is doing the learner a disservice.
The reason this problem is sharper for a model is that the model can be consistently confident-sounding, in a way that no human tutor can. A human tutor who is uncertain shows it in their face, in their pace, in the small hedges that human speech naturally produces. A model produces clean text, and clean text reads as confidence, and confidence reads as knowledge. The model's uncertainty is invisible to the learner unless the model chooses to surface it.
We spent the first year of Socrates shipping the tutor without explicit uncertainty handling, and we saw the cost in the transcripts. A learner would ask a question at the edge of the tutor's competence—say, the exact circumstances under which a particular theorem fails—and the tutor would respond with a fluent paragraph that confidently asserted a slightly wrong edge case. The learner, having no way to know that the edge case was wrong, would incorporate it into their model. Two sessions later, when the learner encountered the real edge case, they would have to unlearn the wrong one. The unlearning is expensive. The original misconception is what is cheap to produce and what we were producing too much of.
Three ways to make uncertainty useful
We tested three response patterns. The first named the boundary and stopped: “I am not confident enough to state that as fact.” The second named the boundary and suggested a check: “Here is the part I would verify before relying on it.” The third named the boundary and asked the learner to choose the next move: “Should we work from the definition, or compare two examples?”
The first pattern was honest but often stalled the session. The second improved verification, especially in technical topics. The third produced the most useful follow-up questions, but only when the tutor had already made clear what it did know. Uncertainty without a foothold felt like abandonment; uncertainty attached to a next step felt like collaboration.
Each pattern is a different way of converting the model's internal uncertainty into something the learner can act on. The honest pause converts uncertainty into time: the learner gets a moment to consider whether they, too, are uncertain, and the tutor gets a moment to think. The boundary statement converts uncertainty into epistemic structure: the learner is shown, in plain language, where the model's knowledge ends and its guesswork begins. The ask-the-user pattern converts uncertainty into collaboration: the tutor treats the learner as a source of evidence rather than a recipient of answers.
The three patterns are not mutually exclusive. In a single difficult session, a tutor might honestly pause, then state the boundary, then ask the user what they have already ruled out. We have found that the order matters: the honest pause should come first, because it buys the model the time it needs to actually consider the boundary. The boundary statement should come second, because it is most useful after the model has had time to think. The ask-the-user should come last, because it is most useful after the learner has seen the boundary and can now contribute more pointedly.
The point of saying “I am unsure” is not to end the explanation. It is to make the next move visible.
Error-catching by subject, with and without a named boundary
What changed in the tutor
The current tutor separates three signals that used to be blended together: uncertainty about a fact, uncertainty about the learner’s meaning, and uncertainty about which example would help. Each one now leads to a different move. A factual boundary points toward a source or a check. A meaning boundary points toward a clarifying question. An example boundary points toward a smaller case.
We also removed a phrase that sounded responsible but did not help: “As an AI, I may be wrong.” It is true, but it gives the learner no information about where the risk is or what to do next. Specific uncertainty is more useful than a general disclaimer.
The "boundary + next step" finding is one we did not expect to be so clean. We had expected the next step to be a small additional lift, on top of the lift from the boundary statement. We did not expect the next step to be a large additional lift, on the order of 37 percentage points over the boundary-only condition. The most plausible interpretation is that a next step gives the learner something to do, and "something to do" is the prerequisite for a follow-up question. A learner who has been told where the tutor's knowledge ends, but who has not been told what to try next, is a learner who has to invent the next move. A learner who has been told where the knowledge ends and what to try next is a learner who has been handed the next move.
Specific uncertainty is more useful than a general disclaimer. Name the edge, then point at the next step.
What the data says
The 62 reviewed sessions produced 214 answer episodes where a factual error was possible. The tutor named a confidence boundary before answering in about half of them:
Answer episodes · 214
Feb 2026 · math, programming, historyThe pattern that changed our design was the last row. Naming a boundary and stopping produced honest answers but stalled sessions; naming a boundary and attaching a next step—a source, a check, or a smaller case—produced the follow-up questions that make a tutoring conversation useful.
Which pattern, exactly
The three response patterns split cleanly in the transcripts. Stopping is honest but inert; attaching a next step keeps the session alive; handing the choice to the learner produces the best follow-ups at a small cost in verification.
| Response pattern | Session continued | Useful follow-up | Learner verified |
|---|---|---|---|
| Boundary named, then stop | 44% | 31% | 38% |
| Boundary + suggested check | 78% | 52% | 61% |
| Boundary + choice of next move | 85% | 64% | 57% |
Table 1 — Outcomes by response pattern · 107 boundary-naming episodes · Feb 2026
How we measured it
Sessions were sampled from February 2026 production logs across mathematics, programming, and history, then anonymised. Two reviewers coded each answer episode for whether a boundary was named, whether a next step was attached, and whether the learner caught any error or produced a follow-up. Disagreements were resolved by a third reviewer; the pattern held in all three subjects, with the largest effect in programming.
The largest effect in programming is, on reflection, unsurprising. Programming is the subject in which the boundary conditions are most clearly defined and most often wrong. A learner who is learning about, say, Python's GIL, or JavaScript's event loop, or Rust's borrow checker, is a learner who is encountering precise boundaries that the model can easily fudge. The fudging reads as confidence, and the confidence reads as knowledge, and the learner ends up with a wrong model that is hard to dislodge. The boundary statement is, in programming, the most valuable single intervention we have.
The pattern also held in history, which surprised us. We had expected history to be the subject in which boundary statements would be least useful, because history is the subject in which most "facts" are actually interpretations. The data says otherwise. Learners who received a boundary statement in a history session produced useful follow-up questions at almost the same rate as learners in programming. The most plausible interpretation is that boundary statements are useful not because they tell the learner where the truth ends, but because they tell the learner where the model's certainty ends. The latter is useful in every subject, even subjects in which the underlying truth is fuzzy.
We used the same three reviewers across the entire study. Each reviewer saw every session. The inter-rater κ was 0.79, which is acceptable for this kind of coding. The lower κ is in part a reflection of the genuinely fuzzy boundary between "the tutor named a boundary" and "the tutor named something that could be interpreted as a boundary." We revised the rubric mid-study to require the boundary to be in a specific syntactic form ("the boundary is," "this stops working when," "I'm not sure about"), which raised the κ to 0.84 in the second half of the study.
Limitations
Three limits are worth naming. Sixty-two sessions and 214 episodes are enough to see a pattern and not enough to calibrate it per subject. The reviewers knew the study's question, and the sessions were sampled from logs rather than randomised. And the boundaries we studied were tutor-selected; whether model-calibrated confidence would draw the same boundaries is an open and, we suspect, more important question.
Open questions
How much boundary-marking is too much? We are tracking return behaviour and follow-up rates, not just correctness. Whether learners come to trust a tutor that is frequently uncertain—or tune it out—needs a longer horizon than 62 sessions. And we have not yet tested the difference between model-calibrated uncertainty and tutor-selected uncertainty: which boundaries to name may matter more than how to phrase them.
The "how much is too much" question is the one we are most worried about. A tutor that names a boundary on every answer is, in effect, a tutor that does not commit to anything. The learner, who came for answers, gets hedges. The hedges are accurate but they are not what the learner wanted. The current production behaviour is to name a boundary only when the model's calibrated uncertainty exceeds a threshold; the threshold is tuned, but it is not principled, and we are aware that we are optimising against a moving target.
The long-term trust question is the one we have least data on. We have some evidence that learners who encounter frequent boundary statements in their first month have higher retention at six months, on the theory that they have built a more accurate model of what the tutor does and does not know. We do not yet know whether that higher retention translates into higher return visits. The next study will measure return visits as a primary outcome.
The calibrated-versus-selected question is the one we have argued about most internally. Model-calibrated uncertainty is the number the model already has—it is the probability the model assigns to its own answer. Tutor-selected uncertainty is the human-curated set of boundaries that the model is told to name. Our current production system is a blend: the model is allowed to name any boundary its calibrated uncertainty exceeds the threshold for, but there is also a small set of boundaries that the system always names, regardless of calibration (the GIL example above is one of these). The blend is pragmatic. We do not yet know whether either pure strategy would be better.
A fourth open question, which we have not yet begun to study: does the value of the boundary statement depend on the learner's prior relationship with the tutor? A learner who has had ten sessions with the tutor may interpret a boundary statement as "the tutor is being careful," while a learner in their first session may interpret the same statement as "the tutor does not know." The framing matters, and we do not yet have data on how to frame.
References
- Graesser, A. C., Person, N., & Magliano, J. (1995). Collaborative dialogue patterns in naturalistic one-to-one tutoring. Applied Cognitive Psychology, 9(6), 495–522.
- Aleven, V., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognition and Instruction, 20(2), 181–238.
- Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980. In Judgment under Uncertainty (pp. 306–334). Cambridge University Press.
- Koriat, A. (1997). Monitoring one's own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23(1), 132–147.