The first question to ask about a tutoring session is not whether the learner understood. It is what, a week later, they still carry.
Key takeaways
- Learners keep the detour, not the destination. Of 186 remembered moments cited a week later, 71% were detours — pauses at an unmotivated step where the learner asked why.
- Polished summaries are forgotten. Only 18% of remembered moments were the clean, complete explanations we used to optimise for.
- Endings changed. We removed the end-of-session summary; the tutor now closes by surfacing open questions instead.
- A minority wants closure. About 15% of participants explicitly preferred a clean summary — a design problem we have not solved yet.
How we ran the study
Between February and April 2026 we asked 47 active learners to participate in a small follow-up study. A week after each learner's session, we sent them a short form with three questions: which moment from the session do you remember most, which moment, if any, has changed how you think about the topic, and is there anything from the session that surprised you. We paired their answers with the transcripts.
The transcripts were, on average, twenty-six minutes long. The median learner had asked between four and seven distinct questions. The recalled moments, almost always, were not the answers.
The choice of "revisit" as the primary signal was deliberate. A learner who voluntarily opens a session from a week ago and asks the tutor to bring back a specific moment is a learner who has, on some level, decided that moment was worth their time. A learner who merely recognises the moment when prompted is a learner who has not made that decision. The first is a measure of what the learner keeps; the second is a measure of what the learner has. The two are not the same, and the difference matters for what we are trying to learn.
A second design choice was the opt-in. We did not want to recruit a random sample of all learners; we wanted learners who were willing to be re-contacted a week after a session and who were willing to spend ten minutes on a follow-up form. The opt-in is a bias, and we are explicit about it: the cohort over-represents active learners, and the effects we report are, if anything, larger than they would be in an unselected sample. We accept the bias because the alternative—random sampling—would have produced a sample with too few revisits to measure.
A third design choice was the form. We sent three questions, not one: which moment from the session do you remember most, which moment, if any, has changed how you think about the topic, and is there anything from the session that surprised you. The three questions are not the same question. The first asks for salience. The second asks for transfer. The third asks for surprise. We wanted all three because we did not yet know which would produce the cleanest signal, and we did not want to commit to one before we had data.
What got remembered
The thing that got remembered most reliably was the detour. A learner would be walking through a worked example, hit a moment where the answer felt unmotivated, and ask a sharp follow-up: “Why this step, and not the other one?” That moment, more than any other in the transcript, was the one they cited a week later.
The second thing was the moment of explicit uncertainty we wrote about in an earlier note. A learner who said "wait, I am confused" and was met with a smaller question, not a re-explanation, reliably remembered the moment of resolution.
The thing that got remembered least reliably was the polished answer. Learners who had received a clean, complete explanation of a topic, with no friction, were the ones whose recall was flattest at the one-week mark. They had understood the topic. They had not done anything with it.
The first pattern is the one that changed our behaviour. We had assumed, going into the study, that the moments learners would revisit would be the moments they had explicitly asked to remember—moments where the tutor had prompted them to save something to their mistake book. The data says otherwise. The moments learners revisited were the moments where the learner had produced a half-formed explanation and then watched the tutor repair it. The mistake book, in other words, was capturing the wrong things. The moments worth capturing were not the moments the tutor had marked; they were the moments the tutor had repaired.
This is a different theory of what a "mistake book" is for. The traditional theory is that a mistake book is a record of errors, a list of things to avoid. The data says the mistake book is, more usefully, a record of seams, a list of places where the learner's model was thin and the tutor's repair was diagnostic. A seam is not an error. It is a place where an error could have formed and didn't, because the tutor caught the gap. Those are the moments the learner revisits.
A concrete example, from a session in March 2026: a learner was working through a problem about recursion in Python. They wrote a recursive function that returned the wrong value for the base case. The tutor did not point out the error; instead, the tutor said, "what does your function return when n is 0?" The learner paused, said "uh, it returns n, which is 0," and then said, "oh, but 0 is not the right base value here, is it." The tutor said, "what would the right base value be?" The learner said "1, because we're counting down from n to 1." The tutor said "yes." The whole exchange took 90 seconds. A week later, the learner revisited that exact 90 seconds and asked the tutor to bring back the moment.
What the learner was not revisiting was the polished explanation that came after—the tutor's clean walk-through of the corrected recursive function, which took about four minutes and which the learner would, in any other context, have read as the "useful" part of the session. The 90-second seam was more useful to the learner than the four-minute repair, and the learner knew it. We did not know it until we asked.
The second pattern—definitional answers rarely revisited—is the one we found most counter-intuitive. A definition that read cleanly should, on the standard theory of memory, be the most memorable kind of content: it is compact, it is well-structured, and it does not require the learner to do any work to interpret it. The data says otherwise. Definitions that read cleanly were the moments the learner skipped on revisit. The most plausible interpretation is that a clean definition has been fully absorbed at the moment of reading, and there is nothing left for the learner to do when they revisit it. A seam, by contrast, is unresolved, and revisiting it is an opportunity to do the resolution.
The third pattern—comparisons revisited more than definitions—is the one that connects most directly to the shape-change finding in the memory-recall note. A comparison prompt ("compare X and Y") is a small structural intervention that requires the learner to elaborate. The elaboration is what makes the moment worth revisiting.
The session that ends in a clear summary is the session that disappears. The session that ends in an unresolved edge is the session that gets remembered.
What learners still carry, by week since the session
How this changed the product
We made three small changes as a result. First, the tutor no longer offers to summarise a session at the end. It used to do this; the data suggested that summaries were actively unhelpful. Second, the tutor now finishes a session by surfacing one or two specific questions the learner raised but did not finish—work that is still open, rather than work that is done. Third, the recall queue now weights open questions more heavily than closed ones, so the next session starts in the middle of an unfinished thought.
None of these are dramatic changes. Together they shift the centre of gravity of a session away from closure and toward continuity, which is what the data suggested we should have been optimising for all along.
The session that ends in an unresolved edge is the session that gets remembered.
What the data says
The 47 participants named 186 remembered moments across their follow-up forms. Almost none of them were the polished answers:
Remembered moments · 186
Feb – Apr 2026 · 47 learnersThe detour finding is the one we keep returning to. A learner who paused at an unmotivated step and asked "why this step, and not the other one?" could reconstruct the whole surrounding idea a week later. The learners who received a frictionless, complete explanation remembered the topic, but could not re-enter it.
Where in the session the memories live
Matching each remembered moment to its transcript position shows the middle of the session does the work. Openings set the frame; endings close it; neither is what a week later still has weight.
| Moment type | Session position | Moments cited | Share of 186 |
|---|---|---|---|
| Detour on a worked example | Middle | 132 | 71% |
| Uncertainty resolved with a smaller question | Middle | 119 | 64% |
| Framing and motivation | Opening | 41 | 22% |
| Polished summary answer | End | 33 | 18% |
Table 1 — Moments could be cited in more than one category · two-coder matching, κ = 0.71
How we measured it
Forty-seven active learners opted in and answered a three-question form one week after their session: the moment they remembered most, the moment that changed how they think about the topic, and anything that surprised them. Two coders matched each answer to a transcript moment; matches were only counted when both coders agreed. The study was internally reviewed but not externally peer-reviewed, and the cohort over-represents active learners.
The matching was the hardest part of the measurement. A learner who writes "the moment where I finally got why the function returns 1" is referring to a specific moment, but the wording does not contain enough information to identify the moment uniquely. The two coders worked independently to match each answer to a transcript moment, and we counted only the matches both coders agreed on. The agreement rate was 76%, which is on the lower end of what we accept; the lower agreement is in part an artefact of the open-ended response format, which allowed learners to describe moments in their own words rather than selecting from a list.
We did not measure retention directly. A learner who revisits a moment is not necessarily a learner who has retained the underlying idea. The revisit is a leading indicator; the retention is the lagging one. We have a small follow-up study planned for September 2026 that will pair revisits with a delayed free-recall prompt, on the model of the memory-recall study.
Limitations
Four limits bound the findings. Forty-seven learners is a interview-scale study, not a survey-scale one. The cohort over-represents highly active learners, who may value detours more than occasional users would. Self-reported memory is the measure itself here—there is no independent recall test to validate it. And the follow-up forms came from us, the people whose product the sessions happened in, which is exactly the kind of framing a skeptical reader should discount for.
Open questions
About 15% of participants explicitly wanted closure, and we have not yet designed an ending that serves both preferences. We also do not know whether ending sessions on open questions fatigues learners over months, or whether the effect generalises beyond highly active learners. Both are on the study list.
The 15% who wanted closure is a real design problem. A learner who wants closure is a learner who experiences an open-ended ending as incomplete. We have tried three responses: a default open-ended ending with an opt-in for a clean summary; a default clean summary with an opt-in for an open-ended ending; and a context-sensitive default that picks open-ended for sessions with detected seams and clean summary for sessions without. None of the three has been formally A/B tested, and we are aware that any of them is a compromise.
The "fatigue over months" question is the one we are most uncertain about. The current behaviour—open-ended endings, seams saved by default—may produce durable learning in the short term and yet fatigue the learner in the long term. A learner who has had fifty sessions of being asked to revisit seams may eventually experience the revisits as a chore. We do not yet know. A twelve-month longitudinal study is on the calendar, but it has not yet started.
The "generalises beyond active learners" question is the one we are most cautious about. Our cohort over-represents active learners, and the effects may not transfer to learners who use the tutor less frequently or who have less prior context. A replication with a broader cohort is planned for the second half of 2026.
A fourth open question, which we have begun to think about: does the revisit behaviour depend on the kind of seam? A seam that surfaced a boundary condition may be more worth revisiting than a seam that surfaced a step-justification gap, on the theory that boundary conditions are more durable features of the topic. We do not yet have data on this, but we are designing the next study to look at it directly.
References
- Murre, J. M. J., & Dros, J. (2015). Replication and analysis of Ebbinghaus' forgetting curve. PLoS ONE, 10(7), e0120644.
- Rubin, D. C., & Wenzel, A. E. (1996). One hundred years of forgetting: A quantitative description of retention. Psychological Review, 103(4), 734–760.
- Hunt, R. R. (2006). The concept of distinctiveness in memory research. In Distinctiveness and Memory (pp. 3–25). Oxford University Press.
- Bjork, R. A., & Bjork, E. L. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In Psychology and the Real World (pp. 56–64). Worth Publishers.