AI Helped Students Learn More—Even After the Chat Closed

The most useful question about AI in education may not be whether students use it. It may be what the tool leaves them able to do after it is gone.

A randomized experiment at Middlebury College offers an unusually concrete answer. Researchers Zara Contractor and Germán Reyes assigned undergraduates to study an unfamiliar topic—blockchain, carbon capture or CRISPR—and write a roughly 500-word analytical essay during a 35-minute learning period. Students were randomly assigned to a condition that allowed generative AI or one that prohibited it. Everyone then took a test without AI or other resources, and 205 students returned about a week later for another unaided test and essay.

Students who had been allowed to use AI scored 7.2 percentage points higher on the immediate knowledge test than the control group, whose average was 56.3%. The analysis included 210 students with immediate test data. A week later, the difference was smaller but still positive: 3.9 percentage points above a control mean of 50.0%. In standardized terms, the estimated intent-to-treat effects were 0.285 standard deviations immediately and 0.213 standard deviations later. The delayed estimate met the authors’ 10% significance threshold, not the conventional 5% threshold.

This does not mean every way of using AI improves learning. The experiment establishes an average effect of being allowed access under these conditions. Its conversation-log evidence offers a separate, descriptive look at what students did with that access.

Research references: [1]

Learning rather than merely producing

The study separated assisted output from retained knowledge. During the first session, AI-allowed students could use the tool while preparing their essays, but the knowledge tests were closed-resource. In the second session, about a week later, everyone completed both the test and essay without AI or outside material. Students had not been told in advance that they would return for those assessments.

Access to AI changed how students spent the fixed learning period. Compared with the control group, students allowed to use AI reported shifting 5.3 percentage points of their time away from writing activities and 4.4 points toward reading and searching. Total time on task did not change. Enjoyment rose by 0.58 points on a 0–10 scale, an estimate with p=0.063.

Conversation logs also showed how varied “AI use” can be. About 70% of the AI-allowed group used generative AI. Explaining concepts was the most common logged use, followed by drafting, editing and summarizing. A research assistant later classified AI users’ conversations as augmentation, automation, mixed or off-topic. Among users, 61% were classified as pure augmentation, 9% as pure automation, 27% as mixed and 3% as other.

Those categories were based on students’ choices, not randomized assignments, so differences among them cannot show that one mode caused better learning. They are still useful as a hypothesis. Augmentation users scored 0.345 standard deviations above the control group on the immediate test and 0.243 standard deviations higher about a week later. Automation users submitted much stronger essays while AI was available—1.498 standard deviations above the control group on overall quality—but that advantage was absent when they wrote unaided later. Their later test and essay estimates were negative and imprecise.

The immediate product could therefore look better while the person’s durable capability did not. That pattern is descriptive, but it gives educators and designers a question worth testing directly: does the interface invite the learner to think with the system, or simply make thinking easier to bypass?

Research references: [1]

A design principle for human agency

This is not an argument for banning generation. It is a reason to judge educational AI by a better standard than the quality of the output produced while the tool is open.

A system that explains an unfamiliar idea, answers follow-up questions, challenges a learner’s reasoning or critiques a draft can increase the return on the learner’s own effort. A system that completes the task may remove some of the practice through which knowledge is formed. The same model can support either relationship. Interface choices, assignments and incentives help shape which relationship a learner adopts.

For educators, that suggests pairing AI-enabled work with moments of unaided performance. Ask students to explain the reasoning in their own words, apply it to a new case or return to the subject after a delay. Assess what persists, not only what was submitted. For product designers, it suggests building for dialogue, retrieval practice and revision rather than treating the fastest completed answer as the sole measure of success.

The study sets important boundaries. It was conducted at one selective U.S. college in spring 2025, with 211 students attending the first session and 205 completing both. AI use at the college already exceeded 80%, so participants were not novices encountering the technology for the first time. The learning period was proctored, capped at 35 minutes and reinforced by financial incentives. Three assigned topics and a one-week horizon cannot establish what happens across a semester, at other institutions or in everyday study.

The authors also disclosed four early protocol deviations: the first session’s verbal script omitted the AI condition, one timer malfunctioned, the first follow-up slot lacked a virtual waiting room and one follow-up slot had an essay-platform glitch. They used session fixed effects to account for those variations. Return rates did not differ significantly between conditions, and 97.2% of first-session participants returned.

The study was registered as AEARCTR-0015806 before the experiment, with an initial registration date of April 14, 2025. It received Middlebury institutional review board approval under protocol 447 and support from the college’s Office of the Provost. The current public research article is arXiv v2, revised October 4, 2026, rather than a peer-reviewed journal publication.

Still, the result points beyond a false choice between rejecting AI and surrendering to it. AI can make the work easier without making the learner less necessary. The better test is whether a person emerges with more understanding, more confidence to act and more capacity after the assistance ends.

That is a humane standard for educational technology—and perhaps for AI at work more broadly. The best tools do not merely produce on our behalf. They help us become more able to choose, judge and create for ourselves.

: Zara Contractor and Germán Reyes, “Experimental Evidence on the Learning Impact of Generative AI,” arXiv v2, October 4, 2026. https://arxiv.org/pdf/2607.08849 : AEA RCT Registry, AEARCTR-0015806, “Experimental Evidence on the Learning Impact of Generative AI.” https://www.socialscienceregistry.org/trials/15806 : arXiv record for 2607.08849, version history and abstract. https://arxiv.org/abs/2607.08849

Research references: [1] [2] [3]